2022-07-18T02:54:03.699 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-18T02:54:03.702 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_mbenjamin@shatner --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} --name mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi --verbose --archive /home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300 -- /home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300/orig.config.yaml 2022-07-18T02:54:03.752 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-18T02:54:03.825 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300 branch: wip-rgwlc-azone description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '6935300' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-rgwlc-azone ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: ff10064ab878e58afb1d2538b2086a2d4a41c017 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: ff10064ab878e58afb1d2538b2086a2d4a41c017 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-rgwlc-azone sha1: ff10064ab878e58afb1d2538b2086a2d4a41c017 owner: scheduled_mbenjamin@shatner priority: 70 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: ff10064ab878e58afb1d2538b2086a2d4a41c017 sleep_before_teardown: 0 suite: rgw suite_branch: wip-rgwlc-azone suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ff10064ab878e58afb1d2538b2086a2d4a41c017 targets: smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbz643aJzTA1MGouCBCaX2wAlr7gxhsuzbiWpPfB1J+XwHbAt+kDOGp2Ft/jOzaXFKrq9e20ENTcd5lzC730MsuhxkXAFpXApXKyIfdGdowsKOGictcs22h7dXfvGGl7lytZg65AHT5kAGQZkADb24rTTvlM9ZtQrM2wLMo5pHBM4p+2nlHFEMH4NvWM0W4qQ+MUTv+V9EbLfB3Kf4OJcfMuw4S9+fSDf3bqVb8I7WIx/wtYFxEEJT4PolE/dvR7kCe/C6rqqqDMP154xs5/17c652VftNZq41Vv8IueKmd3wq2ZOM2zqrNLCGr1FkE7T4jBC89zWOCn9PRu+RF9Vl0lPLKOrkVOVOoQdEZSZixPiOEkRyKEkWNxkyIPtS/EDTjjDKIZSrCeRLm3EOXMP0eFTEa98qWrsoaxtLOeDIPM3HwufNX5YU5DLyvmx+x9ZQilp3QwJD89g+dzQqSSdDgxjTvJ2ZDvid4MkWd8G5Faw/UCUtHNq8FLc9BCGZ+pM= smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0XX/4wSzWw+RBIBAqfnofJBXzIjtevgLHZtgd1wszCkZ2NrfTAt7OZWuj/Imft/uz6GkoqE0v5k/VRYEThvU4Fg0FCIKaBPOc7rZOR3Ego4Qk+Xun+Fm0AfDMCVtZTsCY/dImi9Kh2bOruLyGZnRPD4hnLoYNgsHioSMJjGOyWj5cIzGU7u/UNmmfSDvQqTnBUbvXYWLuRkwYDnyUGt36BVOM+OHhS3ODwnKXtNycReRYuxdoh8T4Q193sdYXaaISZxtQnPiIG/tRMy7vDanx5m+m9I0i65kh/J4umQQuqQ/DutpXO8TrPXAxqYtkyloh9IrBRV1bwN55ziIA9cPLHl9LBFA7cGKjk/2Bm5tEz9DxeMWUPygu6jFc+UUh3zNTaaJp9nSPMYmwYP8Ub7TmEluVDqPJh6jctHSYllsA/scLg19rIGTi7E1s7KxLOu8bBT++U9T6HPeVXqwZqZlLmEoIBKTE2wnIS5NHj/M/pdzfAvd/OqdUsOJGxMTJHDc= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: args: - -a - '!bucket_sync_disable,!bucket_reshard' config: reconfigure_delay: 60 teuthology_branch: main teuthology_sha1: 89d874d555f56b1c1f383df72487aecec0ee6cf9 timestamp: 2022-07-17_22:24:15 tube: smithi user: mbenjamin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-18T02:54:03.825 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa; will attempt to use it 2022-07-18T02:54:03.826 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa/tasks 2022-07-18T02:54:03.826 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-18T02:54:03.829 INFO:teuthology.task.internal:Checking packages... 2022-07-18T02:54:03.858 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ff10064ab878e58afb1d2538b2086a2d4a41c017' 2022-07-18T02:54:03.859 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-18T02:54:03.859 INFO:teuthology.packaging:ref: None 2022-07-18T02:54:03.860 INFO:teuthology.packaging:tag: None 2022-07-18T02:54:03.860 INFO:teuthology.packaging:branch: wip-rgwlc-azone 2022-07-18T02:54:03.860 INFO:teuthology.packaging:sha1: ff10064ab878e58afb1d2538b2086a2d4a41c017 2022-07-18T02:54:03.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rgwlc-azone 2022-07-18T02:54:04.029 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-13595.gff10064a 2022-07-18T02:54:04.029 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-18T02:54:04.046 INFO:teuthology.task.internal:no buildpackages task found 2022-07-18T02:54:04.047 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-18T02:54:04.058 INFO:teuthology.task.internal:Saving configuration 2022-07-18T02:54:04.104 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-18T02:54:04.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-18T02:54:04.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-18 02:48:41.615333', 'locked_by': 'scheduled_mbenjamin@shatner', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDS9RlFkrNbnjphFseTxwxyM3PSFhS/8sF4O9v8UpX2Dch1B8T7h/lH008Nj3ELTxl3U3eR1rwBNl/dkr8PbSgs='} 2022-07-18T02:54:04.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-18 02:48:41.613830', 'locked_by': 'scheduled_mbenjamin@shatner', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNX32MpuUaooSjpqHJSPom9L/sKpWy3bJa7wO9dKVpVqEvTWMUbelN6bjCtwNxcmnzGlc+s2/AI4s2RqS/C5sEE='} 2022-07-18T02:54:04.178 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-18T02:54:04.191 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-07-18T02:54:04.192 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-07-18T02:54:04.192 INFO:teuthology.run_tasks:Running task console_log... 2022-07-18T02:54:04.355 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdcadc92d08>, signals=[15]) 2022-07-18T02:54:04.357 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-18T02:54:04.372 INFO:teuthology.task.internal:Opening connections... 2022-07-18T02:54:04.373 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2022-07-18T02:54:04.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-18T02:54:04.508 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2022-07-18T02:54:04.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-18T02:54:04.656 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-18T02:54:04.669 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2022-07-18T02:54:04.691 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2022-07-18T02:54:04.691 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2022-07-18T02:54:04.749 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2022-07-18T02:54:04.750 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8" 2022-07-18T02:54:04.750 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2022-07-18T02:54:04.750 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2022-07-18T02:54:04.750 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8" 2022-07-18T02:54:04.751 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2022-07-18T02:54:04.751 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-18T02:54:04.751 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2022-07-18T02:54:04.751 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-18T02:54:04.752 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2022-07-18T02:54:04.752 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-18T02:54:04.752 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-18T02:54:04.752 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-18T02:54:04.753 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2022-07-18T02:54:04.780 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2022-07-18T02:54:04.801 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2022-07-18T02:54:04.802 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2022-07-18T02:54:04.862 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2022-07-18T02:54:04.862 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8" 2022-07-18T02:54:04.863 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2022-07-18T02:54:04.863 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2022-07-18T02:54:04.863 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8" 2022-07-18T02:54:04.863 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2022-07-18T02:54:04.864 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-18T02:54:04.864 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2022-07-18T02:54:04.864 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-18T02:54:04.864 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2022-07-18T02:54:04.865 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-18T02:54:04.865 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-18T02:54:04.865 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-18T02:54:04.865 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2022-07-18T02:54:04.886 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-18T02:54:04.904 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-18T02:54:04.917 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-18T02:54:04.918 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2022-07-18T02:54:04.922 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2022-07-18T02:54:04.944 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-18T02:54:04.954 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-18T02:54:04.955 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2022-07-18T02:54:04.982 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2022-07-18T02:54:05.012 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-18T02:54:05.076 INFO:teuthology.run_tasks:Running task kernel... 2022-07-18T02:54:05.102 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-07-18T02:54:05.103 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-07-18T02:54:05.104 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-18T02:54:05.106 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-18T02:54:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T02:54:05.126 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2022-07-18T02:54:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T02:54:05.128 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2022-07-18T02:54:05.188 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-394.el8.x86_64 2022-07-18T02:54:05.188 INFO:teuthology.task.kernel:Running kernel on smithi133: 4.18.0-394.el8.x86_64 2022-07-18T02:54:05.189 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2022-07-18T02:54:05.191 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-394.el8.x86_64 2022-07-18T02:54:05.192 INFO:teuthology.task.kernel:Running kernel on smithi159: 4.18.0-394.el8.x86_64 2022-07-18T02:54:05.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2022-07-18T02:54:08.584 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - AppStream 28 MB/s | 24 MB 00:00 2022-07-18T02:54:08.671 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - AppStream 25 MB/s | 24 MB 00:00 2022-07-18T02:54:13.406 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - BaseOS 36 MB/s | 23 MB 00:00 2022-07-18T02:54:13.559 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - BaseOS 36 MB/s | 23 MB 00:00 2022-07-18T02:54:16.989 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - Extras 161 kB/s | 18 kB 00:00 2022-07-18T02:54:17.174 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - Extras 184 kB/s | 18 kB 00:00 2022-07-18T02:54:17.358 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - PowerTools 19 MB/s | 4.9 MB 00:00 2022-07-18T02:54:17.543 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - PowerTools 19 MB/s | 4.9 MB 00:00 2022-07-18T02:54:20.258 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-07-18T02:54:20.276 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 2.0 kB/s | 3.5 kB 00:01 2022-07-18T02:54:21.359 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 13 MB/s | 12 MB 00:00 2022-07-18T02:54:21.375 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 12 MB/s | 12 MB 00:01 2022-07-18T02:54:24.224 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 608 kB/s | 24 kB 00:00 2022-07-18T02:54:24.243 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 430 kB/s | 24 kB 00:00 2022-07-18T02:54:26.888 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-07-18T02:54:26.889 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-394.el8.x86_64 is already installed. 2022-07-18T02:54:26.974 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-07-18T02:54:26.974 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-394.el8.x86_64 is already installed. 2022-07-18T02:54:27.019 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T02:54:27.020 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T02:54:27.020 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T02:54:27.093 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T02:54:27.094 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T02:54:27.094 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T02:54:27.184 DEBUG:teuthology.orchestra.run.smithi159:> echo no | sudo yum reinstall kernel || true 2022-07-18T02:54:27.231 DEBUG:teuthology.orchestra.run.smithi133:> echo no | sudo yum reinstall kernel || true 2022-07-18T02:54:27.729 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:03 ago on Mon 18 Jul 2022 02:54:24 AM UTC. 2022-07-18T02:54:27.769 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:03 ago on Mon 18 Jul 2022 02:54:24 AM UTC. 2022-07-18T02:54:28.699 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T02:54:28.699 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:54:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2022-07-18T02:54:28.700 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:54:28.701 INFO:teuthology.orchestra.run.smithi159.stdout:Reinstalling: 2022-07-18T02:54:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 4.18.0-394.el8 CentOS-BaseOS 8.3 M 2022-07-18T02:54:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:54:28.702 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T02:54:28.703 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:54:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:54:28.704 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 8.3 M 2022-07-18T02:54:28.704 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 0 2022-07-18T02:54:28.705 INFO:teuthology.orchestra.run.smithi159.stderr:Operation aborted. 2022-07-18T02:54:28.743 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T02:54:28.743 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:54:28.744 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2022-07-18T02:54:28.744 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:54:28.745 INFO:teuthology.orchestra.run.smithi133.stdout:Reinstalling: 2022-07-18T02:54:28.745 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 4.18.0-394.el8 CentOS-BaseOS 8.3 M 2022-07-18T02:54:28.745 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:54:28.746 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T02:54:28.746 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:54:28.747 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:54:28.747 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 8.3 M 2022-07-18T02:54:28.747 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 0 2022-07-18T02:54:28.748 INFO:teuthology.orchestra.run.smithi133.stderr:Operation aborted. 2022-07-18T02:54:28.768 INFO:teuthology.orchestra.run.smithi159.stdout:Is this ok [y/N]: 2022-07-18T02:54:28.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum reinstall -y kernel || true 2022-07-18T02:54:28.801 INFO:teuthology.orchestra.run.smithi133.stdout:Is this ok [y/N]: 2022-07-18T02:54:28.802 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum reinstall -y kernel || true 2022-07-18T02:54:29.297 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:05 ago on Mon 18 Jul 2022 02:54:24 AM UTC. 2022-07-18T02:54:29.331 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:05 ago on Mon 18 Jul 2022 02:54:24 AM UTC. 2022-07-18T02:54:30.268 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T02:54:30.269 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:54:30.269 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2022-07-18T02:54:30.278 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:54:30.279 INFO:teuthology.orchestra.run.smithi159.stdout:Reinstalling: 2022-07-18T02:54:30.279 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 4.18.0-394.el8 CentOS-BaseOS 8.3 M 2022-07-18T02:54:30.279 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:54:30.279 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T02:54:30.280 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:54:30.280 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:54:30.280 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 8.3 M 2022-07-18T02:54:30.281 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 0 2022-07-18T02:54:30.281 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T02:54:30.289 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T02:54:30.289 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:54:30.289 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2022-07-18T02:54:30.289 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:54:30.290 INFO:teuthology.orchestra.run.smithi133.stdout:Reinstalling: 2022-07-18T02:54:30.290 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 4.18.0-394.el8 CentOS-BaseOS 8.3 M 2022-07-18T02:54:30.291 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:54:30.291 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T02:54:30.292 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:54:30.293 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:54:30.293 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 8.3 M 2022-07-18T02:54:30.294 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 0 2022-07-18T02:54:30.294 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T02:54:30.664 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-394.el8.x86_64.rpm 22 MB/s | 8.3 MB 00:00 2022-07-18T02:54:30.664 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:54:30.665 INFO:teuthology.orchestra.run.smithi133.stdout:Total 22 MB/s | 8.3 MB 00:00 2022-07-18T02:54:30.665 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-394.el8.x86_64.rpm 21 MB/s | 8.3 MB 00:00 2022-07-18T02:54:30.665 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:54:30.666 INFO:teuthology.orchestra.run.smithi159.stdout:Total 21 MB/s | 8.3 MB 00:00 2022-07-18T02:54:30.750 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T02:54:30.751 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T02:54:31.139 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T02:54:31.139 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T02:54:31.141 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T02:54:31.141 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T02:54:31.304 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T02:54:31.305 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T02:54:31.307 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T02:54:31.309 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T02:54:31.586 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T02:54:31.613 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T02:54:31.794 INFO:teuthology.orchestra.run.smithi159.stdout: Reinstalling : kernel-4.18.0-394.el8.x86_64 1/2 2022-07-18T02:54:31.795 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : kernel-4.18.0-394.el8.x86_64 2/2 2022-07-18T02:54:31.845 INFO:teuthology.orchestra.run.smithi133.stdout: Reinstalling : kernel-4.18.0-394.el8.x86_64 1/2 2022-07-18T02:54:31.845 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : kernel-4.18.0-394.el8.x86_64 2/2 2022-07-18T02:54:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-4.18.0-394.el8.x86_64 2/2 2022-07-18T02:54:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-4.18.0-394.el8.x86_64 1/2 2022-07-18T02:54:32.221 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-4.18.0-394.el8.x86_64 2/2 2022-07-18T02:54:32.221 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-4.18.0-394.el8.x86_64 1/2 2022-07-18T02:54:32.584 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-4.18.0-394.el8.x86_64 2/2 2022-07-18T02:54:32.585 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:54:32.585 INFO:teuthology.orchestra.run.smithi159.stdout:Reinstalled: 2022-07-18T02:54:32.585 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-4.18.0-394.el8.x86_64 2022-07-18T02:54:32.586 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:54:32.586 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T02:54:32.678 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel --last | head -n 1 2022-07-18T02:54:32.680 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-4.18.0-394.el8.x86_64 2/2 2022-07-18T02:54:32.685 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:54:32.686 INFO:teuthology.orchestra.run.smithi133.stdout:Reinstalled: 2022-07-18T02:54:32.687 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-4.18.0-394.el8.x86_64 2022-07-18T02:54:32.687 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:54:32.688 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T02:54:32.755 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel --last | head -n 1 2022-07-18T02:54:32.813 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-394.el8.x86_64 Mon 18 Jul 2022 02:54:31 AM UTC 2022-07-18T02:54:32.815 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-394.el8.x86_64 2022-07-18T02:54:32.815 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-18T02:54:32.815 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-07-18T02:54:32.816 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-18T02:54:32.847 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2022-07-18T02:54:32.861 DEBUG:teuthology.parallel:result is None 2022-07-18T02:54:32.904 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-394.el8.x86_64 Mon 18 Jul 2022 02:54:31 AM UTC 2022-07-18T02:54:32.905 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-394.el8.x86_64 2022-07-18T02:54:32.905 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-18T02:54:32.905 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-07-18T02:54:32.906 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-18T02:54:32.942 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2022-07-18T02:54:32.953 DEBUG:teuthology.parallel:result is None 2022-07-18T02:54:32.954 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-18T02:54:32.969 INFO:teuthology.task.internal:Creating test directory... 2022-07-18T02:54:32.970 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-18T02:54:32.999 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-18T02:54:33.019 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-18T02:54:33.031 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-18T02:54:33.065 INFO:teuthology.task.internal:Creating archive directory... 2022-07-18T02:54:33.068 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-18T02:54:33.093 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-18T02:54:33.140 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-18T02:54:33.157 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-18T02:54:33.158 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-18T02:54:33.176 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-18T02:54:33.204 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-18T02:54:33.214 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-18T02:54:33.228 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-18T02:54:33.238 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-18T02:54:33.240 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-18T02:54:33.256 INFO:teuthology.task.internal:Configuring sudo... 2022-07-18T02:54:33.257 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-18T02:54:33.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-18T02:54:33.299 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-18T02:54:33.309 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-18T02:54:33.310 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-18T02:54:33.339 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-18T02:54:33.357 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-18T02:54:33.460 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-18T02:54:33.493 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-18T02:54:33.592 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-18T02:54:33.636 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T02:54:33.637 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-18T02:54:33.708 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-18T02:54:33.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-18T02:54:33.807 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-18T02:54:33.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-18T02:54:33.953 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T02:54:33.953 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-18T02:54:34.021 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2022-07-18T02:54:34.024 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2022-07-18T02:54:34.094 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-07-18T02:54:34.123 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-07-18T02:54:34.488 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-18T02:54:34.540 INFO:teuthology.task.internal:Starting timer... 2022-07-18T02:54:34.540 INFO:teuthology.run_tasks:Running task pcp... 2022-07-18T02:54:34.587 INFO:teuthology.run_tasks:Running task selinux... 2022-07-18T02:54:34.608 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-07-18T02:54:34.609 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2022-07-18T02:54:34.678 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2022-07-18T02:54:34.678 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2022-07-18T02:54:34.736 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2022-07-18T02:54:34.738 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-18T02:54:34.738 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2022-07-18T02:54:34.780 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2022-07-18T02:54:34.780 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2022-07-18T02:54:34.804 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2022-07-18T02:54:34.805 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi133.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2022-07-18T02:54:34.805 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-07-18T02:54:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T02:54:34.866 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-07-18T02:54:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T02:54:34.934 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-18T02:54:34.934 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2022-07-18T02:54:34.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2022-07-18T02:54:35.002 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-18T02:54:35.026 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-07-18T02:54:35.028 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-18T02:54:35.199 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-18T02:54:35.235 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi133.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-18T02:58:31.367 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi133.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2022-07-18T02:58:31.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2022-07-18T02:58:31.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-18T02:58:31.476 DEBUG:teuthology.orchestra.run.smithi133:> true 2022-07-18T02:58:31.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2022-07-18T02:58:31.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-18T02:58:31.631 DEBUG:teuthology.orchestra.run.smithi159:> true 2022-07-18T02:58:31.702 INFO:teuthology.run_tasks:Running task clock... 2022-07-18T02:58:31.716 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-07-18T02:58:31.716 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-18T02:58:31.720 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-18T02:58:31.722 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-18T02:58:31.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-18T02:58:31.751 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-07-18T02:58:31.768 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-07-18T02:58:31.781 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-07-18T02:58:31.788 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-07-18T02:58:31.830 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2022-07-18T02:58:31.840 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2022-07-18T02:58:31.853 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2022-07-18T02:58:31.857 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-07-18T02:58:31.862 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2022-07-18T02:58:31.868 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-07-18T02:58:31.876 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-07-18T02:58:31.889 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-07-18T02:58:31.931 INFO:teuthology.orchestra.run.smithi133.stderr:bash: ntpq: command not found 2022-07-18T02:58:31.935 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-07-18T02:58:31.936 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2022-07-18T02:58:31.936 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.937 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.937 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.938 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.954 INFO:teuthology.orchestra.run.smithi159.stderr:bash: ntpq: command not found 2022-07-18T02:58:31.956 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-07-18T02:58:31.957 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2022-07-18T02:58:31.957 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.957 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.958 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.958 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-18T02:58:31.967 INFO:teuthology.run_tasks:Running task install... 2022-07-18T02:58:31.991 DEBUG:teuthology.task.install:project ceph 2022-07-18T02:58:31.992 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017'}} 2022-07-18T02:58:31.992 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017'} 2022-07-18T02:58:31.993 INFO:teuthology.task.install:Using flavor: default 2022-07-18T02:58:32.005 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-07-18T02:58:32.006 INFO:teuthology.task.install:extra packages: [] 2022-07-18T02:58:32.007 DEBUG:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2022-07-18T02:58:32.043 INFO:teuthology.orchestra.run.smithi133.stdout:CentOSStream 2022-07-18T02:58:32.044 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-07-18T02:58:32.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo lsb_release -is 2022-07-18T02:58:32.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'tag': None, 'wait_for_package': False} 2022-07-18T02:58:32.050 DEBUG:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2022-07-18T02:58:32.089 INFO:teuthology.orchestra.run.smithi159.stdout:CentOSStream 2022-07-18T02:58:32.089 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-07-18T02:58:32.090 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'tag': None, 'wait_for_package': False} 2022-07-18T02:58:32.090 DEBUG:teuthology.orchestra.run.smithi159:> sudo lsb_release -is 2022-07-18T02:58:32.127 INFO:teuthology.orchestra.run.smithi133.stdout:CentOSStream 2022-07-18T02:58:32.128 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-07-18T02:58:32.129 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ff10064ab878e58afb1d2538b2086a2d4a41c017 2022-07-18T02:58:32.179 INFO:teuthology.orchestra.run.smithi159.stdout:CentOSStream 2022-07-18T02:58:32.179 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-07-18T02:58:32.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ff10064ab878e58afb1d2538b2086a2d4a41c017 2022-07-18T02:58:32.300 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/ 2022-07-18T02:58:32.301 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13595.gff10064a 2022-07-18T02:58:32.348 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/ 2022-07-18T02:58:32.349 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13595.gff10064a 2022-07-18T02:58:32.434 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-07-18T02:58:32.435 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T02:58:32.435 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-07-18T02:58:32.465 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-07-18T02:58:32.466 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ff10064ab878e58afb1d2538b2086a2d4a41c017/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-07-18T02:58:32.476 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwlc-azone/ff10064ab878e58afb1d2538b2086a2d4a41c017/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-07-18T02:58:32.476 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T02:58:32.477 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-07-18T02:58:32.508 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-07-18T02:58:32.509 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ff10064ab878e58afb1d2538b2086a2d4a41c017/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-07-18T02:58:32.536 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-07-18T02:58:32.577 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-07-18T02:58:32.614 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-07-18T02:58:32.655 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-07-18T02:58:32.681 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2022-07-18T02:58:32.683 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2022-07-18T02:58:32.723 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2022-07-18T02:58:32.727 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2022-07-18T02:58:32.921 INFO:teuthology.orchestra.run.smithi133.stdout:55 files removed 2022-07-18T02:58:32.945 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw 2022-07-18T02:58:32.971 INFO:teuthology.orchestra.run.smithi159.stdout:55 files removed 2022-07-18T02:58:32.994 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw 2022-07-18T02:58:33.571 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 284 kB/s | 80 kB 00:00 2022-07-18T02:58:33.613 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 287 kB/s | 80 kB 00:00 2022-07-18T02:58:33.855 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 38 kB/s | 9.3 kB 00:00 2022-07-18T02:58:33.897 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 38 kB/s | 9.3 kB 00:00 2022-07-18T02:58:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 6.3 kB/s | 1.5 kB 00:00 2022-07-18T02:58:34.175 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 5.9 kB/s | 1.5 kB 00:00 2022-07-18T02:58:34.840 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - AppStream 34 MB/s | 24 MB 00:00 2022-07-18T02:58:34.934 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - AppStream 32 MB/s | 24 MB 00:00 2022-07-18T02:58:39.708 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - BaseOS 38 MB/s | 23 MB 00:00 2022-07-18T02:58:39.784 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - BaseOS 36 MB/s | 23 MB 00:00 2022-07-18T02:58:43.383 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - Extras 101 kB/s | 18 kB 00:00 2022-07-18T02:58:43.425 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - Extras 122 kB/s | 18 kB 00:00 2022-07-18T02:58:43.752 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-8 - PowerTools 19 MB/s | 4.9 MB 00:00 2022-07-18T02:58:43.795 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-8 - PowerTools 19 MB/s | 4.9 MB 00:00 2022-07-18T02:58:46.843 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 1.7 kB/s | 3.5 kB 00:02 2022-07-18T02:58:46.900 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2022-07-18T02:58:48.593 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 12 MB 00:01 2022-07-18T02:58:49.250 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 12 MB 00:02 2022-07-18T02:58:51.527 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 310 kB/s | 24 kB 00:00 2022-07-18T02:58:52.209 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 394 kB/s | 24 kB 00:00 2022-07-18T02:58:54.466 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T02:58:54.469 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:58:54.470 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T02:58:54.470 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:58:54.470 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T02:58:54.471 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:17.0.0-13595.gff10064a.el8 ceph 17 M 2022-07-18T02:58:54.471 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2022-07-18T02:58:54.472 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.6 M 2022-07-18T02:58:54.472 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.9 M 2022-07-18T02:58:54.473 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T02:58:54.473 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:17.0.0-13595.gff10064a.el8 ceph 5.3 M 2022-07-18T02:58:54.474 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:17.0.0-13595.gff10064a.el8 ceph 24 M 2022-07-18T02:58:54.474 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:17.0.0-13595.gff10064a.el8 ceph 24 k 2022-07-18T02:58:54.475 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-07-18T02:58:54.475 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 798 k 2022-07-18T02:58:54.475 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-07-18T02:58:54.476 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-07-18T02:58:54.476 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 484 k 2022-07-18T02:58:54.477 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-07-18T02:58:54.484 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 5.5 M 2022-07-18T02:58:54.485 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-07-18T02:58:54.485 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-07-18T02:58:54.486 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse 2022-07-18T02:58:54.487 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 2:17.0.0-13595.gff10064a.el8 ceph 45 k 2022-07-18T02:58:54.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:17.0.0-13595.gff10064a.el8 ceph 107 k 2022-07-18T02:58:54.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:17.0.0-13595.gff10064a.el8 ceph 178 k 2022-07-18T02:58:54.488 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-07-18T02:58:54.490 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:17.0.0-13595.gff10064a.el8 ceph 366 k 2022-07-18T02:58:54.491 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:17.0.0-13595.gff10064a.el8 ceph 345 k 2022-07-18T02:58:54.491 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:17.0.0-13595.gff10064a.el8 ceph 110 k 2022-07-18T02:58:54.492 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-07-18T02:58:54.492 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:58:54.493 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T02:58:54.493 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:58:54.494 INFO:teuthology.orchestra.run.smithi159.stdout:Install 21 Packages 2022-07-18T02:58:54.494 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 2 Packages 2022-07-18T02:58:54.495 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:58:54.495 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 64 M 2022-07-18T02:58:54.495 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T02:58:55.161 INFO:teuthology.orchestra.run.smithi159.stdout:(1/23): ceph-base-17.0.0-13595.gff10064a.el8.x8 7.6 MB/s | 5.3 MB 00:00 2022-07-18T02:58:55.169 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T02:58:55.172 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:58:55.172 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T02:58:55.172 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:58:55.172 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T02:58:55.173 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:17.0.0-13595.gff10064a.el8 ceph 17 M 2022-07-18T02:58:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2022-07-18T02:58:55.173 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.6 M 2022-07-18T02:58:55.173 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.9 M 2022-07-18T02:58:55.174 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T02:58:55.174 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:17.0.0-13595.gff10064a.el8 ceph 5.3 M 2022-07-18T02:58:55.174 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:17.0.0-13595.gff10064a.el8 ceph 24 M 2022-07-18T02:58:55.174 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:17.0.0-13595.gff10064a.el8 ceph 24 k 2022-07-18T02:58:55.175 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-07-18T02:58:55.175 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 798 k 2022-07-18T02:58:55.175 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-07-18T02:58:55.175 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-07-18T02:58:55.175 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 484 k 2022-07-18T02:58:55.176 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-07-18T02:58:55.176 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:17.0.0-13595.gff10064a.el8 ceph 5.5 M 2022-07-18T02:58:55.176 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-07-18T02:58:55.176 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-07-18T02:58:55.177 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse 2022-07-18T02:58:55.177 INFO:teuthology.orchestra.run.smithi133.stdout: x86_64 2:17.0.0-13595.gff10064a.el8 ceph 45 k 2022-07-18T02:58:55.177 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:17.0.0-13595.gff10064a.el8 ceph 107 k 2022-07-18T02:58:55.177 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:17.0.0-13595.gff10064a.el8 ceph 178 k 2022-07-18T02:58:55.177 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-07-18T02:58:55.178 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:17.0.0-13595.gff10064a.el8 ceph 366 k 2022-07-18T02:58:55.178 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:17.0.0-13595.gff10064a.el8 ceph 345 k 2022-07-18T02:58:55.178 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:17.0.0-13595.gff10064a.el8 ceph 110 k 2022-07-18T02:58:55.178 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-07-18T02:58:55.179 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:58:55.179 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T02:58:55.179 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:58:55.179 INFO:teuthology.orchestra.run.smithi133.stdout:Install 21 Packages 2022-07-18T02:58:55.180 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2022-07-18T02:58:55.180 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:58:55.181 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 64 M 2022-07-18T02:58:55.182 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T02:58:55.245 INFO:teuthology.orchestra.run.smithi159.stdout:(2/23): ceph-selinux-17.0.0-13595.gff10064a.el8 290 kB/s | 24 kB 00:00 2022-07-18T02:58:55.411 INFO:teuthology.orchestra.run.smithi159.stdout:(3/23): libcephfs2-17.0.0-13595.gff10064a.el8.x 4.7 MB/s | 798 kB 00:00 2022-07-18T02:58:55.558 INFO:teuthology.orchestra.run.smithi159.stdout:(4/23): libradosstriper1-17.0.0-13595.gff10064a 3.8 MB/s | 484 kB 00:00 2022-07-18T02:58:56.463 INFO:teuthology.orchestra.run.smithi159.stdout:(5/23): ceph-common-17.0.0-13595.gff10064a.el8. 12 MB/s | 24 MB 00:01 2022-07-18T02:58:56.554 INFO:teuthology.orchestra.run.smithi159.stdout:(6/23): librgw2-17.0.0-13595.gff10064a.el8.x86_ 5.4 MB/s | 5.5 MB 00:01 2022-07-18T02:58:56.579 INFO:teuthology.orchestra.run.smithi159.stdout:(7/23): python3-ceph-argparse-17.0.0-13595.gff1 382 kB/s | 45 kB 00:00 2022-07-18T02:58:56.630 INFO:teuthology.orchestra.run.smithi159.stdout:(8/23): python3-ceph-common-17.0.0-13595.gff100 1.4 MB/s | 107 kB 00:00 2022-07-18T02:58:56.655 INFO:teuthology.orchestra.run.smithi159.stdout:(9/23): python3-cephfs-17.0.0-13595.gff10064a.e 2.3 MB/s | 178 kB 00:00 2022-07-18T02:58:56.738 INFO:teuthology.orchestra.run.smithi159.stdout:(10/23): python3-rbd-17.0.0-13595.gff10064a.el8 4.1 MB/s | 345 kB 00:00 2022-07-18T02:58:56.763 INFO:teuthology.orchestra.run.smithi159.stdout:(11/23): python3-rados-17.0.0-13595.gff10064a.e 2.7 MB/s | 366 kB 00:00 2022-07-18T02:58:56.789 INFO:teuthology.orchestra.run.smithi159.stdout:(12/23): python3-rgw-17.0.0-13595.gff10064a.el8 2.1 MB/s | 110 kB 00:00 2022-07-18T02:58:56.877 INFO:teuthology.orchestra.run.smithi159.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 3.1 MB/s | 354 kB 00:00 2022-07-18T02:58:56.906 INFO:teuthology.orchestra.run.smithi159.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2022-07-18T02:58:56.931 INFO:teuthology.orchestra.run.smithi159.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2022-07-18T02:58:56.948 INFO:teuthology.orchestra.run.smithi159.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2022-07-18T02:58:57.071 INFO:teuthology.orchestra.run.smithi133.stdout:(1/23): ceph-base-17.0.0-13595.gff10064a.el8.x8 2.8 MB/s | 5.3 MB 00:01 2022-07-18T02:58:57.107 INFO:teuthology.orchestra.run.smithi159.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 372 kB/s | 59 kB 00:00 2022-07-18T02:58:57.126 INFO:teuthology.orchestra.run.smithi133.stdout:(2/23): ceph-selinux-17.0.0-13595.gff10064a.el8 444 kB/s | 24 kB 00:00 2022-07-18T02:58:57.235 INFO:teuthology.orchestra.run.smithi133.stdout:(3/23): libcephfs2-17.0.0-13595.gff10064a.el8.x 7.2 MB/s | 798 kB 00:00 2022-07-18T02:58:57.316 INFO:teuthology.orchestra.run.smithi159.stdout:(18/23): ceph-radosgw-17.0.0-13595.gff10064a.el 5.9 MB/s | 17 MB 00:02 2022-07-18T02:58:57.318 INFO:teuthology.orchestra.run.smithi133.stdout:(4/23): libradosstriper1-17.0.0-13595.gff10064a 5.7 MB/s | 484 kB 00:00 2022-07-18T02:58:57.341 INFO:teuthology.orchestra.run.smithi159.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 321 kB/s | 75 kB 00:00 2022-07-18T02:58:57.366 INFO:teuthology.orchestra.run.smithi159.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 734 kB/s | 306 kB 00:00 2022-07-18T02:58:57.742 INFO:teuthology.orchestra.run.smithi159.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 4.1 MB/s | 1.7 MB 00:00 2022-07-18T02:58:57.883 INFO:teuthology.orchestra.run.smithi159.stdout:(22/23): librados2-17.0.0-13595.gff10064a.el8.x 6.7 MB/s | 3.6 MB 00:00 2022-07-18T02:58:57.943 INFO:teuthology.orchestra.run.smithi159.stdout:(23/23): librbd1-17.0.0-13595.gff10064a.el8.x86 6.7 MB/s | 3.9 MB 00:00 2022-07-18T02:58:57.946 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:58:57.946 INFO:teuthology.orchestra.run.smithi159.stdout:Total 18 MB/s | 64 MB 00:03 2022-07-18T02:58:57.985 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T02:58:58.039 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T02:58:58.039 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T02:58:58.085 INFO:teuthology.orchestra.run.smithi133.stdout:(5/23): librgw2-17.0.0-13595.gff10064a.el8.x86_ 7.2 MB/s | 5.5 MB 00:00 2022-07-18T02:58:58.135 INFO:teuthology.orchestra.run.smithi133.stdout:(6/23): python3-ceph-argparse-17.0.0-13595.gff1 898 kB/s | 45 kB 00:00 2022-07-18T02:58:58.202 INFO:teuthology.orchestra.run.smithi133.stdout:(7/23): python3-ceph-common-17.0.0-13595.gff100 1.6 MB/s | 107 kB 00:00 2022-07-18T02:58:58.278 INFO:teuthology.orchestra.run.smithi133.stdout:(8/23): python3-cephfs-17.0.0-13595.gff10064a.e 2.3 MB/s | 178 kB 00:00 2022-07-18T02:58:58.361 INFO:teuthology.orchestra.run.smithi133.stdout:(9/23): python3-rados-17.0.0-13595.gff10064a.el 4.3 MB/s | 366 kB 00:00 2022-07-18T02:58:58.462 INFO:teuthology.orchestra.run.smithi133.stdout:(10/23): python3-rbd-17.0.0-13595.gff10064a.el8 3.4 MB/s | 345 kB 00:00 2022-07-18T02:58:58.537 INFO:teuthology.orchestra.run.smithi133.stdout:(11/23): python3-rgw-17.0.0-13595.gff10064a.el8 1.4 MB/s | 110 kB 00:00 2022-07-18T02:58:58.595 INFO:teuthology.orchestra.run.smithi133.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2022-07-18T02:58:58.637 INFO:teuthology.orchestra.run.smithi133.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2022-07-18T02:58:58.671 INFO:teuthology.orchestra.run.smithi133.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2022-07-18T02:58:58.700 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T02:58:58.703 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T02:58:58.704 INFO:teuthology.orchestra.run.smithi133.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2022-07-18T02:58:58.905 INFO:teuthology.orchestra.run.smithi133.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2022-07-18T02:58:58.972 INFO:teuthology.orchestra.run.smithi133.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 881 kB/s | 59 kB 00:00 2022-07-18T02:58:59.022 INFO:teuthology.orchestra.run.smithi133.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2022-07-18T02:58:59.272 INFO:teuthology.orchestra.run.smithi133.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2022-07-18T02:58:59.903 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T02:59:00.105 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-07-18T02:59:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-07-18T02:59:00.414 INFO:teuthology.orchestra.run.smithi133.stdout:(20/23): ceph-common-17.0.0-13595.gff10064a.el8 4.5 MB/s | 24 MB 00:05 2022-07-18T02:59:00.509 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-07-18T02:59:00.914 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-07-18T02:59:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:17.0.0-13595.gff10064a.el8.x86_64 4/25 2022-07-18T02:59:01.259 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:17.0.0-13595.gff10064a.el8.x86_64 4/25 2022-07-18T02:59:01.381 INFO:teuthology.orchestra.run.smithi133.stdout:(21/23): librbd1-17.0.0-13595.gff10064a.el8.x86 4.0 MB/s | 3.9 MB 00:00 2022-07-18T02:59:01.493 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 5/25 2022-07-18T02:59:01.658 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 6/25 2022-07-18T02:59:01.673 INFO:teuthology.orchestra.run.smithi133.stdout:(22/23): librados2-17.0.0-13595.gff10064a.el8.x 1.5 MB/s | 3.6 MB 00:02 2022-07-18T02:59:02.084 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 6/25 2022-07-18T02:59:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 7/25 2022-07-18T02:59:02.339 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 7/25 2022-07-18T02:59:02.557 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-07-18T02:59:02.742 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-07-18T02:59:03.287 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-07-18T02:59:03.758 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 10/25 2022-07-18T02:59:03.829 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 10/25 2022-07-18T02:59:03.977 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:17.0.0-13595.gff10064a.el8 11/25 2022-07-18T02:59:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 12/25 2022-07-18T02:59:04.349 INFO:teuthology.orchestra.run.smithi133.stdout:(23/23): ceph-radosgw-17.0.0-13595.gff10064a.el 1.8 MB/s | 17 MB 00:09 2022-07-18T02:59:04.351 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:59:04.352 INFO:teuthology.orchestra.run.smithi133.stdout:Total 6.9 MB/s | 64 MB 00:09 2022-07-18T02:59:04.391 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T02:59:04.435 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 13/25 2022-07-18T02:59:04.443 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T02:59:04.444 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T02:59:04.669 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 14/25 2022-07-18T02:59:04.816 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_ 15/25 2022-07-18T02:59:04.887 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_ 15/25 2022-07-18T02:59:05.113 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2022-07-18T02:59:05.123 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T02:59:05.126 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T02:59:05.300 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2022-07-18T02:59:05.482 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-07-18T02:59:05.650 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:17.0.0-13595.gff10064a.el8.x 19/25 2022-07-18T02:59:06.293 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T02:59:06.453 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-07-18T02:59:06.705 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-07-18T02:59:06.896 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-07-18T02:59:07.306 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-07-18T02:59:07.505 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:17.0.0-13595.gff10064a.el8.x86_64 4/25 2022-07-18T02:59:07.598 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:17.0.0-13595.gff10064a.el8.x86_64 4/25 2022-07-18T02:59:07.863 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 5/25 2022-07-18T02:59:08.029 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 6/25 2022-07-18T02:59:08.442 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 6/25 2022-07-18T02:59:08.597 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 7/25 2022-07-18T02:59:08.662 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 7/25 2022-07-18T02:59:08.886 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-07-18T02:59:09.030 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-07-18T02:59:09.404 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 20/25 2022-07-18T02:59:09.584 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-07-18T02:59:09.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 20/25 2022-07-18T02:59:09.787 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 10/25 2022-07-18T02:59:09.851 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 10/25 2022-07-18T02:59:10.007 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:17.0.0-13595.gff10064a.el8 11/25 2022-07-18T02:59:10.199 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 12/25 2022-07-18T02:59:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 20/25 2022-07-18T02:59:10.442 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 13/25 2022-07-18T02:59:10.454 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 21/25 2022-07-18T02:59:10.530 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 21/25 2022-07-18T02:59:10.656 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 14/25 2022-07-18T02:59:10.712 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 22/25 2022-07-18T02:59:10.820 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_ 15/25 2022-07-18T02:59:10.890 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_ 15/25 2022-07-18T02:59:11.106 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2022-07-18T02:59:11.263 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2022-07-18T02:59:11.444 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-07-18T02:59:11.613 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:17.0.0-13595.gff10064a.el8.x 19/25 2022-07-18T02:59:15.537 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 20/25 2022-07-18T02:59:15.737 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 20/25 2022-07-18T02:59:16.345 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 20/25 2022-07-18T02:59:16.617 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 21/25 2022-07-18T02:59:16.702 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 21/25 2022-07-18T02:59:16.876 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 22/25 2022-07-18T02:59:29.357 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 22/25 2022-07-18T02:59:29.358 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2022-07-18T02:59:29.358 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2022-07-18T02:59:29.359 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2022-07-18T02:59:29.359 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /run 2022-07-18T02:59:29.360 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2022-07-18T02:59:29.360 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2022-07-18T02:59:29.361 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2022-07-18T02:59:29.361 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2022-07-18T02:59:29.361 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2022-07-18T02:59:29.362 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:31.101 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 23/25 2022-07-18T02:59:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 23/25 2022-07-18T02:59:31.174 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-07-18T02:59:31.423 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-07-18T02:59:31.423 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-07-18T02:59:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-07-18T02:59:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 1/25 2022-07-18T02:59:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 2/25 2022-07-18T02:59:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 3/25 2022-07-18T02:59:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 4/25 2022-07-18T02:59:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 5/25 2022-07-18T02:59:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_ 6/25 2022-07-18T02:59:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 7/25 2022-07-18T02:59:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13595.gff10064a.el8 8/25 2022-07-18T02:59:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:17.0.0-13595.gff10064a.el8.x 9/25 2022-07-18T02:59:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 10/25 2022-07-18T02:59:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 11/25 2022-07-18T02:59:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 12/25 2022-07-18T02:59:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 13/25 2022-07-18T02:59:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-07-18T02:59:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-07-18T02:59:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-07-18T02:59:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-07-18T02:59:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-07-18T02:59:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-07-18T02:59:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-07-18T02:59:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-07-18T02:59:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:17.0.0-13595.gff10064a.el8.x86_64 22/25 2022-07-18T02:59:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-07-18T02:59:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 24/25 2022-07-18T02:59:32.728 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-07-18T02:59:32.728 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:32.728 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2022-07-18T02:59:32.729 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.729 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.729 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T02:59:32.729 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.730 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.730 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.730 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.730 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-07-18T02:59:32.731 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.731 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.2-3.el8.x86_64 2022-07-18T02:59:32.731 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-07-18T02:59:32.731 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.732 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-07-18T02:59:32.732 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.732 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.3.1-3.el8.x86_64 2022-07-18T02:59:32.732 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-07-18T02:59:32.732 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.733 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.733 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.733 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-07-18T02:59:32.733 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.734 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.734 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:32.734 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.13.0-2.el8.x86_64 2022-07-18T02:59:32.734 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:32.735 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T02:59:32.957 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test 2022-07-18T02:59:33.530 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:42 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T02:59:34.524 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T02:59:34.525 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:59:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T02:59:34.525 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:59:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T02:59:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:17.0.0-13595.gff10064a.el8 ceph 53 M 2022-07-18T02:59:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T02:59:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-07-18T02:59:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:17.0.0-13595.gff10064a.el8 ceph 170 k 2022-07-18T02:59:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-07-18T02:59:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-07-18T02:59:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-07-18T02:59:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:34.529 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T02:59:34.530 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T02:59:34.530 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2022-07-18T02:59:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:34.532 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 54 M 2022-07-18T02:59:34.533 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 241 M 2022-07-18T02:59:34.534 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T02:59:34.597 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2022-07-18T02:59:34.656 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2022-07-18T02:59:34.715 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2022-07-18T02:59:34.811 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): libcephsqlite-17.0.0-13595.gff10064a.el8 608 kB/s | 170 kB 00:00 2022-07-18T02:59:34.856 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 513 kB/s | 69 kB 00:00 2022-07-18T02:59:35.044 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 22/25 2022-07-18T02:59:35.044 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2022-07-18T02:59:35.045 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2022-07-18T02:59:35.045 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /dev 2022-07-18T02:59:35.045 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /run 2022-07-18T02:59:35.046 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2022-07-18T02:59:35.046 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2022-07-18T02:59:35.047 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2022-07-18T02:59:35.047 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2022-07-18T02:59:35.048 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /dev 2022-07-18T02:59:35.053 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:36.776 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 23/25 2022-07-18T02:59:36.855 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 23/25 2022-07-18T02:59:36.862 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-07-18T02:59:37.079 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-07-18T02:59:37.080 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-07-18T02:59:37.322 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): ceph-test-17.0.0-13595.gff10064a.el8.x86 19 MB/s | 53 MB 00:02 2022-07-18T02:59:37.325 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:59:37.325 INFO:teuthology.orchestra.run.smithi159.stdout:Total 19 MB/s | 54 MB 00:02 2022-07-18T02:59:37.363 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T02:59:37.382 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T02:59:37.383 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T02:59:37.688 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T02:59:37.691 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T02:59:37.753 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-07-18T02:59:37.754 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 1/25 2022-07-18T02:59:37.754 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 2/25 2022-07-18T02:59:37.755 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 3/25 2022-07-18T02:59:37.756 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 4/25 2022-07-18T02:59:37.757 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 5/25 2022-07-18T02:59:37.758 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_ 6/25 2022-07-18T02:59:37.758 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 7/25 2022-07-18T02:59:37.759 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13595.gff10064a.el8 8/25 2022-07-18T02:59:37.759 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:17.0.0-13595.gff10064a.el8.x 9/25 2022-07-18T02:59:37.759 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 10/25 2022-07-18T02:59:37.759 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 11/25 2022-07-18T02:59:37.760 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 12/25 2022-07-18T02:59:37.760 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 13/25 2022-07-18T02:59:37.761 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-07-18T02:59:37.761 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-07-18T02:59:37.761 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-07-18T02:59:37.761 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-07-18T02:59:37.762 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-07-18T02:59:37.762 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-07-18T02:59:37.762 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-07-18T02:59:37.762 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-07-18T02:59:37.763 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:17.0.0-13595.gff10064a.el8.x86_64 22/25 2022-07-18T02:59:37.763 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-07-18T02:59:37.763 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 24/25 2022-07-18T02:59:38.153 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T02:59:38.372 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2022-07-18T02:59:38.430 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-07-18T02:59:38.431 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:38.431 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2022-07-18T02:59:38.432 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.432 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.433 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T02:59:38.433 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.434 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.434 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.435 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.436 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-07-18T02:59:38.439 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.440 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.2-3.el8.x86_64 2022-07-18T02:59:38.451 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-07-18T02:59:38.456 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.456 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-07-18T02:59:38.456 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.457 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.3.1-3.el8.x86_64 2022-07-18T02:59:38.457 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-07-18T02:59:38.458 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.458 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.459 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.459 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-07-18T02:59:38.460 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.460 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.460 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:38.461 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.13.0-2.el8.x86_64 2022-07-18T02:59:38.462 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:38.462 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T02:59:38.548 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-07-18T02:59:38.623 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test 2022-07-18T02:59:38.678 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-07-18T02:59:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-07-18T02:59:39.230 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:47 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T02:59:39.231 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-07-18T02:59:39.412 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 5/6 2022-07-18T02:59:40.222 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T02:59:40.222 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:59:40.223 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T02:59:40.223 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:59:40.223 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T02:59:40.223 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:17.0.0-13595.gff10064a.el8 ceph 53 M 2022-07-18T02:59:40.224 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T02:59:40.224 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-07-18T02:59:40.224 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:17.0.0-13595.gff10064a.el8 ceph 170 k 2022-07-18T02:59:40.224 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-07-18T02:59:40.225 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-07-18T02:59:40.225 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-07-18T02:59:40.225 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:40.225 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T02:59:40.225 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T02:59:40.238 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2022-07-18T02:59:40.239 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:40.240 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 54 M 2022-07-18T02:59:40.241 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 241 M 2022-07-18T02:59:40.241 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T02:59:40.303 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.5 MB/s | 202 kB 00:00 2022-07-18T02:59:40.356 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2022-07-18T02:59:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2022-07-18T02:59:40.556 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 487 kB/s | 69 kB 00:00 2022-07-18T02:59:40.691 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): libcephsqlite-17.0.0-13595.gff10064a.el8 365 kB/s | 170 kB 00:00 2022-07-18T02:59:44.549 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 5/6 2022-07-18T02:59:44.820 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 6/6 2022-07-18T02:59:45.070 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 6/6 2022-07-18T02:59:45.070 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 1/6 2022-07-18T02:59:45.071 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 2/6 2022-07-18T02:59:45.071 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-07-18T02:59:45.071 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-07-18T02:59:45.071 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-07-18T02:59:45.475 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2022-07-18T02:59:45.475 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:45.476 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T02:59:45.476 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:45.477 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-3.el8.x86_64 2022-07-18T02:59:45.478 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:45.479 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-07-18T02:59:45.479 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-1.el8.x86_64 2022-07-18T02:59:45.479 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-07-18T02:59:45.480 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:45.480 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T02:59:45.714 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph 2022-07-18T02:59:46.284 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:55 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T02:59:47.267 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T02:59:47.268 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2022-07-18T02:59:47.268 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T02:59:47.268 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2022-07-18T02:59:47.268 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T02:59:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:17.0.0-13595.gff10064a.el8 ceph 6.5 k 2022-07-18T02:59:47.269 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T02:59:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:17.0.0-13595.gff10064a.el8 ceph 2.3 M 2022-07-18T02:59:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:17.0.0-13595.gff10064a.el8 ceph 1.5 M 2022-07-18T02:59:47.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 230 k 2022-07-18T02:59:47.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.9 M 2022-07-18T02:59:47.280 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:17.0.0-13595.gff10064a.el8 ceph 17 M 2022-07-18T02:59:47.280 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-07-18T02:59:47.281 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-07-18T02:59:47.281 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-07-18T02:59:47.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-07-18T02:59:47.298 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-07-18T02:59:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-07-18T02:59:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-07-18T02:59:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-07-18T02:59:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-07-18T02:59:47.301 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-07-18T02:59:47.301 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-07-18T02:59:47.315 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-07-18T02:59:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-07-18T02:59:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-07-18T02:59:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-07-18T02:59:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-07-18T02:59:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-07-18T02:59:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-07-18T02:59:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-07-18T02:59:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-07-18T02:59:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-07-18T02:59:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-07-18T02:59:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-07-18T02:59:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-07-18T02:59:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-07-18T02:59:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-07-18T02:59:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:47.320 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T02:59:47.320 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2022-07-18T02:59:47.327 INFO:teuthology.orchestra.run.smithi159.stdout:Install 32 Packages 2022-07-18T02:59:47.327 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T02:59:47.328 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 29 M 2022-07-18T02:59:47.328 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 114 M 2022-07-18T02:59:47.328 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T02:59:47.500 INFO:teuthology.orchestra.run.smithi159.stdout:(1/32): ceph-17.0.0-13595.gff10064a.el8.x86_64. 26 kB/s | 6.5 kB 00:00 2022-07-18T02:59:48.229 INFO:teuthology.orchestra.run.smithi159.stdout:(2/32): ceph-mds-17.0.0-13595.gff10064a.el8.x86 2.3 MB/s | 2.3 MB 00:00 2022-07-18T02:59:48.535 INFO:teuthology.orchestra.run.smithi159.stdout:(3/32): ceph-mgr-17.0.0-13595.gff10064a.el8.x86 1.2 MB/s | 1.5 MB 00:01 2022-07-18T02:59:48.724 INFO:teuthology.orchestra.run.smithi159.stdout:(4/32): ceph-mgr-modules-core-17.0.0-13595.gff1 1.2 MB/s | 230 kB 00:00 2022-07-18T02:59:48.806 INFO:teuthology.orchestra.run.smithi159.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-07-18T02:59:48.822 INFO:teuthology.orchestra.run.smithi159.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-07-18T02:59:48.846 INFO:teuthology.orchestra.run.smithi159.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2022-07-18T02:59:48.870 INFO:teuthology.orchestra.run.smithi159.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-07-18T02:59:48.898 INFO:teuthology.orchestra.run.smithi159.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 14 MB/s | 457 kB 00:00 2022-07-18T02:59:48.917 INFO:teuthology.orchestra.run.smithi159.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2022-07-18T02:59:48.951 INFO:teuthology.orchestra.run.smithi159.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2022-07-18T02:59:48.997 INFO:teuthology.orchestra.run.smithi159.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 2.5 MB/s | 109 kB 00:00 2022-07-18T02:59:49.135 INFO:teuthology.orchestra.run.smithi159.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 308 kB/s | 44 kB 00:00 2022-07-18T02:59:49.227 INFO:teuthology.orchestra.run.smithi159.stdout:(14/32): ceph-mon-17.0.0-13595.gff10064a.el8.x8 2.3 MB/s | 3.9 MB 00:01 2022-07-18T02:59:49.279 INFO:teuthology.orchestra.run.smithi159.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2022-07-18T02:59:49.386 INFO:teuthology.orchestra.run.smithi159.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.5 MB/s | 384 kB 00:00 2022-07-18T02:59:49.413 INFO:teuthology.orchestra.run.smithi159.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 939 kB/s | 173 kB 00:00 2022-07-18T02:59:49.447 INFO:teuthology.orchestra.run.smithi159.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2022-07-18T02:59:49.466 INFO:teuthology.orchestra.run.smithi159.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2022-07-18T02:59:49.487 INFO:teuthology.orchestra.run.smithi159.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 969 kB/s | 49 kB 00:00 2022-07-18T02:59:49.512 INFO:teuthology.orchestra.run.smithi159.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2022-07-18T02:59:49.546 INFO:teuthology.orchestra.run.smithi159.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2022-07-18T02:59:49.578 INFO:teuthology.orchestra.run.smithi159.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2022-07-18T02:59:49.593 INFO:teuthology.orchestra.run.smithi159.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 446 kB/s | 19 kB 00:00 2022-07-18T02:59:49.615 INFO:teuthology.orchestra.run.smithi159.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 569 kB/s | 24 kB 00:00 2022-07-18T02:59:49.631 INFO:teuthology.orchestra.run.smithi159.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 694 kB/s | 29 kB 00:00 2022-07-18T02:59:49.648 INFO:teuthology.orchestra.run.smithi159.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 657 kB/s | 27 kB 00:00 2022-07-18T02:59:49.682 INFO:teuthology.orchestra.run.smithi159.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.7 MB/s | 241 kB 00:00 2022-07-18T02:59:49.763 INFO:teuthology.orchestra.run.smithi159.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2022-07-18T02:59:49.789 INFO:teuthology.orchestra.run.smithi159.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 849 kB/s | 85 kB 00:00 2022-07-18T02:59:49.807 INFO:teuthology.orchestra.run.smithi159.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2022-07-18T02:59:49.982 INFO:teuthology.orchestra.run.smithi159.stdout:(32/32): ceph-osd-17.0.0-13595.gff10064a.el8.x8 9.7 MB/s | 17 MB 00:01 2022-07-18T02:59:49.985 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:59:49.986 INFO:teuthology.orchestra.run.smithi159.stdout:Total 10 MB/s | 29 MB 00:02 2022-07-18T02:59:50.057 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T02:59:50.098 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T02:59:50.099 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T02:59:50.433 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T02:59:50.436 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T02:59:51.397 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): ceph-test-17.0.0-13595.gff10064a.el8.x86 4.8 MB/s | 53 MB 00:11 2022-07-18T02:59:51.398 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T02:59:51.398 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.8 MB/s | 54 MB 00:11 2022-07-18T02:59:51.427 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T02:59:51.461 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T02:59:51.461 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T02:59:51.572 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T02:59:51.775 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T02:59:51.778 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T02:59:51.815 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-07-18T02:59:52.017 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-07-18T02:59:52.250 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T02:59:52.278 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-07-18T02:59:52.450 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2022-07-18T02:59:52.474 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-07-18T02:59:52.636 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-07-18T02:59:52.642 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-07-18T02:59:52.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-07-18T02:59:52.808 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-07-18T02:59:52.855 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-07-18T02:59:53.012 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-07-18T02:59:53.029 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-07-18T02:59:53.167 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-07-18T02:59:53.238 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 5/6 2022-07-18T02:59:53.394 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-07-18T02:59:53.659 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-07-18T02:59:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-07-18T02:59:54.128 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-07-18T02:59:54.482 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-07-18T02:59:54.776 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-07-18T02:59:54.950 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-07-18T02:59:55.200 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-07-18T02:59:55.391 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-07-18T02:59:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-07-18T02:59:55.948 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-07-18T02:59:56.089 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-07-18T02:59:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-07-18T02:59:56.548 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-07-18T02:59:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-07-18T02:59:57.019 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-07-18T02:59:57.432 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-07-18T02:59:57.758 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-07-18T02:59:58.106 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13595.gff10064a.el8 27/32 2022-07-18T02:59:58.280 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 28/32 2022-07-18T02:59:58.385 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 5/6 2022-07-18T02:59:58.635 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 6/6 2022-07-18T02:59:58.872 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 6/6 2022-07-18T02:59:58.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 1/6 2022-07-18T02:59:58.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 2/6 2022-07-18T02:59:58.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-07-18T02:59:58.876 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-07-18T02:59:58.876 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-07-18T02:59:59.378 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2022-07-18T02:59:59.378 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:59.380 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T02:59:59.381 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:59.381 INFO:teuthology.orchestra.run.smithi133.stdout: jq-1.6-3.el8.x86_64 2022-07-18T02:59:59.382 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T02:59:59.382 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-07-18T02:59:59.382 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-1.el8.x86_64 2022-07-18T02:59:59.383 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-07-18T02:59:59.383 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T02:59:59.384 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T02:59:59.570 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph 2022-07-18T02:59:59.890 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 28/32 2022-07-18T03:00:00.118 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:08 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:00.306 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 29/32 2022-07-18T03:00:00.700 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 29/32 2022-07-18T03:00:00.864 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 30/32 2022-07-18T03:00:01.078 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 30/32 2022-07-18T03:00:01.099 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:01.102 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2022-07-18T03:00:01.103 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:01.104 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2022-07-18T03:00:01.104 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:01.105 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:17.0.0-13595.gff10064a.el8 ceph 6.5 k 2022-07-18T03:00:01.105 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T03:00:01.106 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:17.0.0-13595.gff10064a.el8 ceph 2.3 M 2022-07-18T03:00:01.106 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:17.0.0-13595.gff10064a.el8 ceph 1.5 M 2022-07-18T03:00:01.107 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 230 k 2022-07-18T03:00:01.107 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.9 M 2022-07-18T03:00:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:17.0.0-13595.gff10064a.el8 ceph 17 M 2022-07-18T03:00:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-07-18T03:00:01.110 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-07-18T03:00:01.110 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-07-18T03:00:01.111 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-07-18T03:00:01.111 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-07-18T03:00:01.112 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-07-18T03:00:01.112 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-07-18T03:00:01.112 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-07-18T03:00:01.113 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-07-18T03:00:01.122 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-07-18T03:00:01.122 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-07-18T03:00:01.123 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-07-18T03:00:01.134 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-07-18T03:00:01.134 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-07-18T03:00:01.136 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-07-18T03:00:01.137 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-07-18T03:00:01.137 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-07-18T03:00:01.137 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-07-18T03:00:01.138 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-07-18T03:00:01.138 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-07-18T03:00:01.139 INFO:teuthology.orchestra.run.smithi133.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-07-18T03:00:01.139 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-07-18T03:00:01.140 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-07-18T03:00:01.140 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-07-18T03:00:01.140 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-07-18T03:00:01.141 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-07-18T03:00:01.141 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:01.142 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:01.143 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2022-07-18T03:00:01.143 INFO:teuthology.orchestra.run.smithi133.stdout:Install 32 Packages 2022-07-18T03:00:01.143 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:01.145 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 29 M 2022-07-18T03:00:01.153 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 114 M 2022-07-18T03:00:01.153 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:01.246 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 31/32 2022-07-18T03:00:01.270 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 31/32 2022-07-18T03:00:01.311 INFO:teuthology.orchestra.run.smithi133.stdout:(1/32): ceph-17.0.0-13595.gff10064a.el8.x86_64. 32 kB/s | 6.5 kB 00:00 2022-07-18T03:00:01.424 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:17.0.0-13595.gff10064a.el8.x86_64 32/32 2022-07-18T03:00:01.627 INFO:teuthology.orchestra.run.smithi133.stdout:(2/32): ceph-mgr-17.0.0-13595.gff10064a.el8.x86 3.0 MB/s | 1.5 MB 00:00 2022-07-18T03:00:01.792 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-2:17.0.0-13595.gff10064a.el8.x86_64 32/32 2022-07-18T03:00:01.793 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:17.0.0-13595.gff10064a.el8.x86_64 1/32 2022-07-18T03:00:01.793 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 2/32 2022-07-18T03:00:01.794 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 3/32 2022-07-18T03:00:01.795 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 4/32 2022-07-18T03:00:01.795 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 5/32 2022-07-18T03:00:01.796 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13595.gff10064a.el8 6/32 2022-07-18T03:00:01.796 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-07-18T03:00:01.797 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-07-18T03:00:01.797 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-07-18T03:00:01.798 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-07-18T03:00:01.798 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-07-18T03:00:01.803 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-07-18T03:00:01.803 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-07-18T03:00:01.804 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-07-18T03:00:01.804 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-07-18T03:00:01.805 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-07-18T03:00:01.805 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-07-18T03:00:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-07-18T03:00:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-07-18T03:00:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-07-18T03:00:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-07-18T03:00:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-07-18T03:00:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-07-18T03:00:01.813 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-07-18T03:00:01.813 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-07-18T03:00:01.814 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-07-18T03:00:01.814 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-07-18T03:00:01.830 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-07-18T03:00:01.830 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-07-18T03:00:01.831 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-07-18T03:00:01.831 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-07-18T03:00:02.161 INFO:teuthology.orchestra.run.smithi133.stdout:(3/32): ceph-mon-17.0.0-13595.gff10064a.el8.x86 4.6 MB/s | 3.9 MB 00:00 2022-07-18T03:00:02.228 INFO:teuthology.orchestra.run.smithi133.stdout:(4/32): ceph-mgr-modules-core-17.0.0-13595.gff1 3.4 MB/s | 230 kB 00:00 2022-07-18T03:00:02.298 INFO:teuthology.orchestra.run.smithi133.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-07-18T03:00:02.342 INFO:teuthology.orchestra.run.smithi133.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 934 kB/s | 39 kB 00:00 2022-07-18T03:00:02.353 INFO:teuthology.orchestra.run.smithi133.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 6.1 MB/s | 103 kB 00:00 2022-07-18T03:00:02.367 INFO:teuthology.orchestra.run.smithi133.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2022-07-18T03:00:02.396 INFO:teuthology.orchestra.run.smithi133.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2022-07-18T03:00:02.438 INFO:teuthology.orchestra.run.smithi133.stdout:(10/32): ceph-mds-17.0.0-13595.gff10064a.el8.x8 1.7 MB/s | 2.3 MB 00:01 2022-07-18T03:00:02.472 INFO:teuthology.orchestra.run.smithi133.stdout:(11/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.1 MB/s | 237 kB 00:00 2022-07-18T03:00:02.501 INFO:teuthology.orchestra.run.smithi133.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.5 MB/s | 109 kB 00:00 2022-07-18T03:00:02.608 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-07-18T03:00:02.608 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:02.608 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:02.609 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:02.609 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:02.609 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:02.609 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:02.610 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:02.610 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:02.611 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-07-18T03:00:02.611 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-07-18T03:00:02.615 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-07-18T03:00:02.616 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-07-18T03:00:02.617 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-07-18T03:00:02.617 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-07-18T03:00:02.617 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-6.2-6.el8.noarch 2022-07-18T03:00:02.618 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-07-18T03:00:02.618 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-07-18T03:00:02.619 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.0.6-13.el8.noarch 2022-07-18T03:00:02.621 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-07-18T03:00:02.622 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-07-18T03:00:02.622 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-07-18T03:00:02.623 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-2.6-1.el8.noarch 2022-07-18T03:00:02.623 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-07-18T03:00:02.624 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.14-14.el8.noarch 2022-07-18T03:00:02.624 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2017.2-9.el8.noarch 2022-07-18T03:00:02.625 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-07-18T03:00:02.625 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-07-18T03:00:02.626 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-07-18T03:00:02.626 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-07-18T03:00:02.626 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-07-18T03:00:02.627 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-07-18T03:00:02.627 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-07-18T03:00:02.628 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-07-18T03:00:02.632 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-07-18T03:00:02.668 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:02.669 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:02.681 INFO:teuthology.orchestra.run.smithi133.stdout:(13/32): ceph-osd-17.0.0-13595.gff10064a.el8.x8 16 MB/s | 17 MB 00:01 2022-07-18T03:00:02.702 INFO:teuthology.orchestra.run.smithi133.stdout:(14/32): python3-cryptography-3.2.1-5.el8.x86_6 2.1 MB/s | 559 kB 00:00 2022-07-18T03:00:02.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base 2022-07-18T03:00:02.860 INFO:teuthology.orchestra.run.smithi133.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 121 kB/s | 44 kB 00:00 2022-07-18T03:00:02.957 INFO:teuthology.orchestra.run.smithi133.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 653 kB/s | 185 kB 00:00 2022-07-18T03:00:02.985 INFO:teuthology.orchestra.run.smithi133.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 608 kB/s | 173 kB 00:00 2022-07-18T03:00:03.035 INFO:teuthology.orchestra.run.smithi133.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2022-07-18T03:00:03.076 INFO:teuthology.orchestra.run.smithi133.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2022-07-18T03:00:03.116 INFO:teuthology.orchestra.run.smithi133.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 583 kB/s | 49 kB 00:00 2022-07-18T03:00:03.167 INFO:teuthology.orchestra.run.smithi133.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 639 kB/s | 59 kB 00:00 2022-07-18T03:00:03.250 INFO:teuthology.orchestra.run.smithi133.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 187 kB/s | 16 kB 00:00 2022-07-18T03:00:03.275 INFO:teuthology.orchestra.run.smithi133.stdout:(23/32): python3-cherrypy-18.4.0-1.el8.noarch.r 918 kB/s | 384 kB 00:00 2022-07-18T03:00:03.300 INFO:teuthology.orchestra.run.smithi133.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2022-07-18T03:00:03.325 INFO:teuthology.orchestra.run.smithi133.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2022-07-18T03:00:03.392 INFO:teuthology.orchestra.run.smithi133.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2022-07-18T03:00:03.397 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:12 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:03.406 INFO:teuthology.orchestra.run.smithi133.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2022-07-18T03:00:03.426 INFO:teuthology.orchestra.run.smithi133.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2022-07-18T03:00:03.485 INFO:teuthology.orchestra.run.smithi133.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2022-07-18T03:00:03.574 INFO:teuthology.orchestra.run.smithi133.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2022-07-18T03:00:04.018 INFO:teuthology.orchestra.run.smithi133.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 144 kB/s | 85 kB 00:00 2022-07-18T03:00:04.043 INFO:teuthology.orchestra.run.smithi133.stdout:(32/32): python3-waitress-1.2.1-2.el8.1.noarch. 361 kB/s | 241 kB 00:00 2022-07-18T03:00:04.048 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:04.048 INFO:teuthology.orchestra.run.smithi133.stdout:Total 9.7 MB/s | 29 MB 00:02 2022-07-18T03:00:04.126 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:04.162 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:04.163 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:04.296 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:00:04.369 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:04.369 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:00:04.370 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:04.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install cephadm 2022-07-18T03:00:04.494 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:04.496 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:04.968 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:13 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:05.682 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:05.877 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-07-18T03:00:05.935 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:05.947 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:05.947 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:05.948 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:05.948 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:05.949 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 79 k 2022-07-18T03:00:05.954 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:05.954 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:05.955 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:05.955 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:00:05.956 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:05.958 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 79 k 2022-07-18T03:00:05.958 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 360 k 2022-07-18T03:00:05.958 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:06.102 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-07-18T03:00:06.123 INFO:teuthology.orchestra.run.smithi159.stdout:cephadm-17.0.0-13595.gff10064a.el8.noarch.rpm 432 kB/s | 79 kB 00:00 2022-07-18T03:00:06.124 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:06.126 INFO:teuthology.orchestra.run.smithi159.stdout:Total 429 kB/s | 79 kB 00:00 2022-07-18T03:00:06.127 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:06.129 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:06.130 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:06.131 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:06.134 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:06.180 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:06.324 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-07-18T03:00:06.507 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-07-18T03:00:06.642 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-07-18T03:00:06.811 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-07-18T03:00:07.034 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-07-18T03:00:07.191 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-07-18T03:00:07.403 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-07-18T03:00:07.652 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-07-18T03:00:07.746 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:07.857 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-07-18T03:00:07.940 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:08.076 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-07-18T03:00:08.124 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:08.434 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-07-18T03:00:08.506 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:08.507 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:08.507 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:08.508 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:08.508 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:08.509 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:08.673 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-07-18T03:00:08.733 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache 2022-07-18T03:00:08.869 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-07-18T03:00:09.102 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-07-18T03:00:09.295 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-07-18T03:00:09.310 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:18 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:09.577 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-07-18T03:00:10.268 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:10.271 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:10.272 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2022-07-18T03:00:10.272 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:10.273 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:10.273 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13595.gff10064a.el8 ceph 170 k 2022-07-18T03:00:10.274 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:10.274 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:10.275 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:10.275 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:00:10.276 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:10.276 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 170 k 2022-07-18T03:00:10.277 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 567 k 2022-07-18T03:00:10.290 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:10.437 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-07-18T03:00:10.507 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-17.0.0-13595.gff100 737 kB/s | 170 kB 00:00 2022-07-18T03:00:10.508 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:10.508 INFO:teuthology.orchestra.run.smithi159.stdout:Total 734 kB/s | 170 kB 00:00 2022-07-18T03:00:10.509 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:10.515 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:10.526 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:10.583 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:10.583 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:10.614 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-07-18T03:00:10.826 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-07-18T03:00:11.067 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-07-18T03:00:11.298 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-07-18T03:00:11.385 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:11.497 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-07-18T03:00:11.900 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-07-18T03:00:11.953 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13595.gff10064a 1/1 2022-07-18T03:00:12.223 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-07-18T03:00:12.352 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13595.gff10064a 1/1 2022-07-18T03:00:12.550 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13595.gff10064a.el8 27/32 2022-07-18T03:00:12.718 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 28/32 2022-07-18T03:00:12.749 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13595.gff10064a 1/1 2022-07-18T03:00:12.750 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:12.750 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:12.751 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:12.751 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:12.751 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:12.928 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr 2022-07-18T03:00:13.487 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:22 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:14.258 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 28/32 2022-07-18T03:00:14.394 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:00:14.414 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 29/32 2022-07-18T03:00:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:00:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:14.557 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-dashboard 2022-07-18T03:00:14.817 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 29/32 2022-07-18T03:00:14.944 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 30/32 2022-07-18T03:00:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:24 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:15.163 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 30/32 2022-07-18T03:00:15.338 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 31/32 2022-07-18T03:00:15.352 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 31/32 2022-07-18T03:00:15.495 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:17.0.0-13595.gff10064a.el8.x86_64 32/32 2022-07-18T03:00:15.903 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-2:17.0.0-13595.gff10064a.el8.x86_64 32/32 2022-07-18T03:00:15.904 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:17.0.0-13595.gff10064a.el8.x86_64 1/32 2022-07-18T03:00:15.904 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 2/32 2022-07-18T03:00:15.905 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 3/32 2022-07-18T03:00:15.905 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 4/32 2022-07-18T03:00:15.905 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 5/32 2022-07-18T03:00:15.906 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13595.gff10064a.el8 6/32 2022-07-18T03:00:15.906 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-07-18T03:00:15.907 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-07-18T03:00:15.907 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-07-18T03:00:15.908 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-07-18T03:00:15.909 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-07-18T03:00:15.909 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-07-18T03:00:15.910 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-07-18T03:00:15.910 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-07-18T03:00:15.910 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-07-18T03:00:15.911 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-07-18T03:00:15.911 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-07-18T03:00:15.912 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-07-18T03:00:15.912 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-07-18T03:00:15.913 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-07-18T03:00:15.913 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-07-18T03:00:15.914 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-07-18T03:00:15.914 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-07-18T03:00:15.914 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-07-18T03:00:15.915 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-07-18T03:00:15.915 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-07-18T03:00:15.916 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-07-18T03:00:15.916 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-07-18T03:00:15.917 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-07-18T03:00:15.917 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-07-18T03:00:15.918 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-07-18T03:00:16.087 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:16.109 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:16.145 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:16.146 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 1.6 M 2022-07-18T03:00:16.159 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T03:00:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards 2022-07-18T03:00:16.163 INFO:teuthology.orchestra.run.smithi159.stdout: noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 23 k 2022-07-18T03:00:16.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 15 k 2022-07-18T03:00:16.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-07-18T03:00:16.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-07-18T03:00:16.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-07-18T03:00:16.165 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:16.166 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:16.166 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:16.167 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2022-07-18T03:00:16.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:16.167 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.9 M 2022-07-18T03:00:16.168 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 7.7 M 2022-07-18T03:00:16.168 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:16.259 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): ceph-grafana-dashboards-17.0.0-13595.gff 149 kB/s | 23 kB 00:00 2022-07-18T03:00:16.288 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): ceph-prometheus-alerts-17.0.0-13595.gff1 82 kB/s | 15 kB 00:00 2022-07-18T03:00:16.321 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 566 kB/s | 43 kB 00:00 2022-07-18T03:00:16.379 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 309 kB/s | 34 kB 00:00 2022-07-18T03:00:16.487 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2022-07-18T03:00:16.598 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): ceph-mgr-dashboard-17.0.0-13595.gff10064 3.2 MB/s | 1.6 MB 00:00 2022-07-18T03:00:16.599 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:16.617 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.8 MB/s | 1.9 MB 00:00 2022-07-18T03:00:16.618 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:16.620 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:16.620 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:16.688 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:16.691 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:16.734 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-07-18T03:00:16.741 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:16.742 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:16.742 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:16.743 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:16.743 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:16.744 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:16.744 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:16.745 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:16.745 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-07-18T03:00:16.754 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-07-18T03:00:16.754 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-07-18T03:00:16.755 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-07-18T03:00:16.755 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-07-18T03:00:16.756 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-07-18T03:00:16.757 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-6.2-6.el8.noarch 2022-07-18T03:00:16.773 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-07-18T03:00:16.774 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-07-18T03:00:16.774 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.0.6-13.el8.noarch 2022-07-18T03:00:16.774 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-07-18T03:00:16.775 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-07-18T03:00:16.775 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-07-18T03:00:16.775 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-2.6-1.el8.noarch 2022-07-18T03:00:16.775 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-07-18T03:00:16.776 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.14-14.el8.noarch 2022-07-18T03:00:16.776 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2017.2-9.el8.noarch 2022-07-18T03:00:16.777 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-07-18T03:00:16.777 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-07-18T03:00:16.777 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-07-18T03:00:16.777 INFO:teuthology.orchestra.run.smithi133.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-07-18T03:00:16.778 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-07-18T03:00:16.778 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-07-18T03:00:16.778 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-07-18T03:00:16.778 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-07-18T03:00:16.779 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-07-18T03:00:16.779 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:16.779 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:16.977 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base 2022-07-18T03:00:17.299 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-07-18T03:00:17.543 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:25 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:18.434 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-07-18T03:00:18.441 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:00:18.510 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:18.514 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:00:18.514 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:18.583 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install cephadm 2022-07-18T03:00:18.615 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-07-18T03:00:18.746 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13595.gff10064a.el8. 4/6 2022-07-18T03:00:19.096 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13595.gff10064a.el8 5/6 2022-07-18T03:00:19.164 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:27 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:19.331 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noar 6/6 2022-07-18T03:00:19.573 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noar 6/6 2022-07-18T03:00:19.573 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13595.gff10064a.el8 1/6 2022-07-18T03:00:19.574 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noar 2/6 2022-07-18T03:00:19.574 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13595.gff10064a.el8. 3/6 2022-07-18T03:00:19.575 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-07-18T03:00:19.575 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-07-18T03:00:19.946 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-07-18T03:00:19.946 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:19.947 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:19.947 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:19.948 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:19.948 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:19.949 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-07-18T03:00:19.949 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-07-18T03:00:19.950 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.4.1-12.el8.noarch 2022-07-18T03:00:19.952 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:19.953 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:20.043 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-diskprediction-local 2022-07-18T03:00:20.128 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:20.136 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:20.137 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:20.137 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:20.138 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:20.138 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 79 k 2022-07-18T03:00:20.139 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:20.139 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:20.140 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:20.140 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:00:20.140 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:20.141 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 79 k 2022-07-18T03:00:20.141 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 360 k 2022-07-18T03:00:20.142 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:20.307 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm-17.0.0-13595.gff10064a.el8.noarch.rpm 430 kB/s | 79 kB 00:00 2022-07-18T03:00:20.324 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:20.325 INFO:teuthology.orchestra.run.smithi133.stdout:Total 427 kB/s | 79 kB 00:00 2022-07-18T03:00:20.326 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:20.327 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:20.328 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:20.328 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:20.329 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:20.362 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:20.613 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:29 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:21.614 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:21.615 INFO:teuthology.orchestra.run.smithi159.stdout:===================================================================================================== 2022-07-18T03:00:21.615 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:21.616 INFO:teuthology.orchestra.run.smithi159.stdout:===================================================================================================== 2022-07-18T03:00:21.616 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:21.617 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 7.4 M 2022-07-18T03:00:21.617 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T03:00:21.618 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 8.5.0-13.el8 CentOS-BaseOS 644 k 2022-07-18T03:00:21.618 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 8.5.0-13.el8 CentOS-BaseOS 171 k 2022-07-18T03:00:21.619 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-07-18T03:00:21.619 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-07-18T03:00:21.620 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-07-18T03:00:21.620 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-07-18T03:00:21.620 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-07-18T03:00:21.621 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:21.621 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:21.623 INFO:teuthology.orchestra.run.smithi159.stdout:===================================================================================================== 2022-07-18T03:00:21.623 INFO:teuthology.orchestra.run.smithi159.stdout:Install 8 Packages 2022-07-18T03:00:21.623 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:21.624 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 35 M 2022-07-18T03:00:21.626 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 210 M 2022-07-18T03:00:21.626 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:21.804 INFO:teuthology.orchestra.run.smithi159.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2022-07-18T03:00:21.864 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:21.938 INFO:teuthology.orchestra.run.smithi159.stdout:(2/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2022-07-18T03:00:22.002 INFO:teuthology.orchestra.run.smithi159.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2022-07-18T03:00:22.027 INFO:teuthology.orchestra.run.smithi159.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2022-07-18T03:00:22.054 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:22.076 INFO:teuthology.orchestra.run.smithi159.stdout:(5/8): libgfortran-8.5.0-13.el8.x86_64.rpm 13 MB/s | 644 kB 00:00 2022-07-18T03:00:22.112 INFO:teuthology.orchestra.run.smithi159.stdout:(6/8): libquadmath-8.5.0-13.el8.x86_64.rpm 5.1 MB/s | 171 kB 00:00 2022-07-18T03:00:22.237 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:22.388 INFO:teuthology.orchestra.run.smithi159.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 35 MB/s | 14 MB 00:00 2022-07-18T03:00:22.628 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:22.628 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:22.629 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:22.629 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:22.630 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:22.631 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:22.814 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache 2022-07-18T03:00:22.952 INFO:teuthology.orchestra.run.smithi159.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-135 5.4 MB/s | 7.4 MB 00:01 2022-07-18T03:00:22.956 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:22.956 INFO:teuthology.orchestra.run.smithi159.stdout:Total 26 MB/s | 35 MB 00:01 2022-07-18T03:00:23.182 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:23.200 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:23.201 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:23.357 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:31 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:23.584 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:23.584 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:24.147 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:24.282 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-8.5.0-13.el8.x86_64 1/8 2022-07-18T03:00:24.306 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:24.306 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:24.307 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2022-07-18T03:00:24.307 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:24.308 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:24.308 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13595.gff10064a.el8 ceph 170 k 2022-07-18T03:00:24.308 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:24.309 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:24.309 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:24.310 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:00:24.310 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:24.310 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 170 k 2022-07-18T03:00:24.311 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 567 k 2022-07-18T03:00:24.311 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:24.402 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libquadmath-8.5.0-13.el8.x86_64 1/8 2022-07-18T03:00:24.542 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-17.0.0-13595.gff100 750 kB/s | 170 kB 00:00 2022-07-18T03:00:24.546 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:24.550 INFO:teuthology.orchestra.run.smithi133.stdout:Total 746 kB/s | 170 kB 00:00 2022-07-18T03:00:24.551 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:24.552 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:24.553 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:24.561 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-8.5.0-13.el8.x86_64 2/8 2022-07-18T03:00:24.593 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:24.594 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:24.724 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:24.900 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13595.gff10064a 1/1 2022-07-18T03:00:25.157 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libgfortran-8.5.0-13.el8.x86_64 2/8 2022-07-18T03:00:25.274 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13595.gff10064a 1/1 2022-07-18T03:00:25.388 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-07-18T03:00:25.633 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13595.gff10064a 1/1 2022-07-18T03:00:25.640 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:25.646 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:25.653 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:25.654 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:25.655 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:25.806 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr 2022-07-18T03:00:25.999 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-07-18T03:00:26.167 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-07-18T03:00:26.359 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:34 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:26.874 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-07-18T03:00:27.215 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-07-18T03:00:27.252 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-mgr-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:00:27.311 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:27.312 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:00:27.312 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:27.438 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-07-18T03:00:27.573 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-dashboard 2022-07-18T03:00:28.181 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:36 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:29.157 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:29.158 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:29.158 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:29.159 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:29.159 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:29.160 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 1.6 M 2022-07-18T03:00:29.160 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T03:00:29.161 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards 2022-07-18T03:00:29.161 INFO:teuthology.orchestra.run.smithi133.stdout: noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 23 k 2022-07-18T03:00:29.162 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 15 k 2022-07-18T03:00:29.163 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-07-18T03:00:29.164 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-07-18T03:00:29.165 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-07-18T03:00:29.166 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:29.167 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:29.168 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:29.168 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2022-07-18T03:00:29.169 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:29.169 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.9 M 2022-07-18T03:00:29.169 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 7.7 M 2022-07-18T03:00:29.170 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-07-18T03:00:29.314 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): ceph-prometheus-alerts-17.0.0-13595.gff1 95 kB/s | 15 kB 00:00 2022-07-18T03:00:29.349 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): ceph-grafana-dashboards-17.0.0-13595.gff 127 kB/s | 23 kB 00:00 2022-07-18T03:00:29.398 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2022-07-18T03:00:29.448 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 309 kB/s | 34 kB 00:00 2022-07-18T03:00:29.551 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): ceph-mgr-dashboard-17.0.0-13595.gff10064 4.2 MB/s | 1.6 MB 00:00 2022-07-18T03:00:29.607 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 941 kB/s | 196 kB 00:00 2022-07-18T03:00:29.608 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:29.609 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.3 MB/s | 1.9 MB 00:00 2022-07-18T03:00:29.619 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:29.628 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:29.628 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:29.710 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:29.711 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:29.954 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:30.180 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-07-18T03:00:30.342 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-07-18T03:00:30.424 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-07-18T03:00:30.494 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-07-18T03:00:30.629 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13595.gff10064a.el8. 4/6 2022-07-18T03:00:30.646 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13595.gff1006 8/8 2022-07-18T03:00:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13595.gff1006 8/8 2022-07-18T03:00:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13595.gff1006 1/8 2022-07-18T03:00:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-07-18T03:00:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-07-18T03:00:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-07-18T03:00:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-07-18T03:00:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-07-18T03:00:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-8.5.0-13.el8.x86_64 7/8 2022-07-18T03:00:30.959 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13595.gff10064a.el8 5/6 2022-07-18T03:00:31.166 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noar 6/6 2022-07-18T03:00:31.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-8.5.0-13.el8.x86_64 8/8 2022-07-18T03:00:31.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:31.317 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:31.317 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:31.317 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-8.5.0-13.el8.x86_64 2022-07-18T03:00:31.321 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-8.5.0-13.el8.x86_64 2022-07-18T03:00:31.321 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.15-3.el8.x86_64 2022-07-18T03:00:31.321 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-07-18T03:00:31.321 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-07-18T03:00:31.322 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-07-18T03:00:31.322 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-07-18T03:00:31.322 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:31.322 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:31.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-rook 2022-07-18T03:00:31.411 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noar 6/6 2022-07-18T03:00:31.412 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13595.gff10064a.el8 1/6 2022-07-18T03:00:31.412 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noar 2/6 2022-07-18T03:00:31.414 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13595.gff10064a.el8. 3/6 2022-07-18T03:00:31.414 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-07-18T03:00:31.415 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-07-18T03:00:31.792 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-07-18T03:00:31.794 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:31.795 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:31.795 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:31.796 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:31.796 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:31.797 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-07-18T03:00:31.799 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-07-18T03:00:31.799 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.4.1-12.el8.noarch 2022-07-18T03:00:31.800 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:31.801 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:31.886 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-diskprediction-local 2022-07-18T03:00:31.957 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:40 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:32.436 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:40 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:32.938 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:32.938 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2022-07-18T03:00:32.939 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:32.939 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2022-07-18T03:00:32.940 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:32.940 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 47 k 2022-07-18T03:00:32.941 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T03:00:32.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-07-18T03:00:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-07-18T03:00:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-07-18T03:00:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-07-18T03:00:32.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-07-18T03:00:32.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-07-18T03:00:32.945 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-07-18T03:00:32.946 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-07-18T03:00:32.946 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-07-18T03:00:32.947 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-07-18T03:00:32.947 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-07-18T03:00:32.948 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-07-18T03:00:32.948 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:32.949 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:32.949 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2022-07-18T03:00:32.949 INFO:teuthology.orchestra.run.smithi159.stdout:Install 13 Packages 2022-07-18T03:00:32.950 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:32.950 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.9 M 2022-07-18T03:00:32.952 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 25 M 2022-07-18T03:00:32.952 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:33.070 INFO:teuthology.orchestra.run.smithi159.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 152 kB/s | 20 kB 00:00 2022-07-18T03:00:33.095 INFO:teuthology.orchestra.run.smithi159.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 169 kB/s | 27 kB 00:00 2022-07-18T03:00:33.122 INFO:teuthology.orchestra.run.smithi159.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2022-07-18T03:00:33.146 INFO:teuthology.orchestra.run.smithi159.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-07-18T03:00:33.194 INFO:teuthology.orchestra.run.smithi159.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.1 MB/s | 155 kB 00:00 2022-07-18T03:00:33.213 INFO:teuthology.orchestra.run.smithi159.stdout:(6/13): ceph-mgr-rook-17.0.0-13595.gff10064a.el 171 kB/s | 47 kB 00:00 2022-07-18T03:00:33.230 INFO:teuthology.orchestra.run.smithi159.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 508 kB/s | 43 kB 00:00 2022-07-18T03:00:33.364 INFO:teuthology.orchestra.run.smithi159.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 108 kB/s | 16 kB 00:00 2022-07-18T03:00:33.368 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:33.369 INFO:teuthology.orchestra.run.smithi133.stdout:===================================================================================================== 2022-07-18T03:00:33.369 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:33.370 INFO:teuthology.orchestra.run.smithi133.stdout:===================================================================================================== 2022-07-18T03:00:33.370 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:33.370 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 7.4 M 2022-07-18T03:00:33.370 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T03:00:33.371 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 8.5.0-13.el8 CentOS-BaseOS 644 k 2022-07-18T03:00:33.371 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 8.5.0-13.el8 CentOS-BaseOS 171 k 2022-07-18T03:00:33.371 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-07-18T03:00:33.372 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-07-18T03:00:33.372 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-07-18T03:00:33.373 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-07-18T03:00:33.373 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-07-18T03:00:33.373 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:33.373 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:33.373 INFO:teuthology.orchestra.run.smithi133.stdout:===================================================================================================== 2022-07-18T03:00:33.374 INFO:teuthology.orchestra.run.smithi133.stdout:Install 8 Packages 2022-07-18T03:00:33.374 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:33.375 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 35 M 2022-07-18T03:00:33.375 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 210 M 2022-07-18T03:00:33.375 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:33.381 INFO:teuthology.orchestra.run.smithi159.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 176 kB/s | 33 kB 00:00 2022-07-18T03:00:33.414 INFO:teuthology.orchestra.run.smithi159.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 442 kB/s | 81 kB 00:00 2022-07-18T03:00:33.466 INFO:teuthology.orchestra.run.smithi159.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-07-18T03:00:33.481 INFO:teuthology.orchestra.run.smithi159.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 613 kB/s | 61 kB 00:00 2022-07-18T03:00:33.553 INFO:teuthology.orchestra.run.smithi133.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2022-07-18T03:00:33.682 INFO:teuthology.orchestra.run.smithi159.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2022-07-18T03:00:33.683 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:33.683 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2022-07-18T03:00:33.703 INFO:teuthology.orchestra.run.smithi133.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2022-07-18T03:00:33.729 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:33.744 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:33.745 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:33.780 INFO:teuthology.orchestra.run.smithi133.stdout:(3/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2022-07-18T03:00:33.804 INFO:teuthology.orchestra.run.smithi133.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2022-07-18T03:00:33.842 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:33.842 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:33.846 INFO:teuthology.orchestra.run.smithi133.stdout:(5/8): libgfortran-8.5.0-13.el8.x86_64.rpm 15 MB/s | 644 kB 00:00 2022-07-18T03:00:33.871 INFO:teuthology.orchestra.run.smithi133.stdout:(6/8): libquadmath-8.5.0-13.el8.x86_64.rpm 6.7 MB/s | 171 kB 00:00 2022-07-18T03:00:34.130 INFO:teuthology.orchestra.run.smithi133.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 39 MB/s | 14 MB 00:00 2022-07-18T03:00:34.906 INFO:teuthology.orchestra.run.smithi133.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-135 4.8 MB/s | 7.4 MB 00:01 2022-07-18T03:00:34.908 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:34.908 INFO:teuthology.orchestra.run.smithi133.stdout:Total 23 MB/s | 35 MB 00:01 2022-07-18T03:00:35.141 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:35.154 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:35.156 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:35.182 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-07-18T03:00:35.505 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:35.507 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:35.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-07-18T03:00:35.830 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-07-18T03:00:36.051 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-07-18T03:00:36.077 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:36.233 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-07-18T03:00:36.253 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-8.5.0-13.el8.x86_64 1/8 2022-07-18T03:00:36.386 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libquadmath-8.5.0-13.el8.x86_64 1/8 2022-07-18T03:00:36.480 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-07-18T03:00:36.498 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-8.5.0-13.el8.x86_64 2/8 2022-07-18T03:00:36.763 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-07-18T03:00:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-07-18T03:00:37.105 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libgfortran-8.5.0-13.el8.x86_64 2/8 2022-07-18T03:00:37.310 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-07-18T03:00:37.930 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-07-18T03:00:38.016 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-07-18T03:00:38.094 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-07-18T03:00:38.281 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-07-18T03:00:38.495 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-07-18T03:00:38.711 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-07-18T03:00:38.777 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-07-18T03:00:38.877 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 13/13 2022-07-18T03:00:39.142 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 13/13 2022-07-18T03:00:39.144 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 1/13 2022-07-18T03:00:39.144 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-07-18T03:00:39.145 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-07-18T03:00:39.145 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-07-18T03:00:39.146 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-07-18T03:00:39.146 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-07-18T03:00:39.147 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-07-18T03:00:39.147 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-07-18T03:00:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-07-18T03:00:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-07-18T03:00:39.149 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-07-18T03:00:39.149 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-07-18T03:00:39.566 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-07-18T03:00:39.566 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:39.567 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:39.569 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:39.569 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-07-18T03:00:39.569 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-07-18T03:00:39.569 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-07-18T03:00:39.570 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-07-18T03:00:39.570 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-07-18T03:00:39.570 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-07-18T03:00:39.570 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-07-18T03:00:39.571 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-07-18T03:00:39.571 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-07-18T03:00:39.571 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-07-18T03:00:39.571 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.8-1.el8.noarch 2022-07-18T03:00:39.571 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-07-18T03:00:39.572 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:39.572 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:39.671 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-cephadm 2022-07-18T03:00:39.771 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-07-18T03:00:39.964 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-07-18T03:00:40.216 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:49 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:41.183 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:41.183 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:41.183 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:41.184 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:41.184 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:41.184 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 111 k 2022-07-18T03:00:41.184 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T03:00:41.185 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-07-18T03:00:41.185 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-07-18T03:00:41.185 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-07-18T03:00:41.185 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-07-18T03:00:41.213 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:41.213 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:41.214 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:41.214 INFO:teuthology.orchestra.run.smithi159.stdout:Install 5 Packages 2022-07-18T03:00:41.214 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:41.214 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 5.9 M 2022-07-18T03:00:41.215 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2022-07-18T03:00:41.215 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:41.383 INFO:teuthology.orchestra.run.smithi159.stdout:(1/5): ceph-mgr-cephadm-17.0.0-13595.gff10064a. 546 kB/s | 111 kB 00:00 2022-07-18T03:00:41.600 INFO:teuthology.orchestra.run.smithi159.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2022-07-18T03:00:41.626 INFO:teuthology.orchestra.run.smithi159.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.9 MB/s | 476 kB 00:00 2022-07-18T03:00:41.734 INFO:teuthology.orchestra.run.smithi159.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 451 kB/s | 60 kB 00:00 2022-07-18T03:00:41.744 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-07-18T03:00:41.811 INFO:teuthology.orchestra.run.smithi159.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 7.7 MB/s | 4.8 MB 00:00 2022-07-18T03:00:41.812 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:41.812 INFO:teuthology.orchestra.run.smithi159.stdout:Total 9.5 MB/s | 5.9 MB 00:00 2022-07-18T03:00:41.853 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:41.864 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:41.865 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:41.961 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:41.961 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:42.239 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:42.592 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-07-18T03:00:42.967 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-07-18T03:00:43.138 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13595.gff1006 8/8 2022-07-18T03:00:43.429 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13595.gff1006 8/8 2022-07-18T03:00:43.429 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13595.gff1006 1/8 2022-07-18T03:00:43.465 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-07-18T03:00:43.466 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-07-18T03:00:43.466 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-07-18T03:00:43.467 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-07-18T03:00:43.467 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-07-18T03:00:43.468 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-8.5.0-13.el8.x86_64 7/8 2022-07-18T03:00:43.505 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-07-18T03:00:43.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-8.5.0-13.el8.x86_64 8/8 2022-07-18T03:00:43.842 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:43.842 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:43.842 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:43.842 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-8.5.0-13.el8.x86_64 2022-07-18T03:00:43.843 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-8.5.0-13.el8.x86_64 2022-07-18T03:00:43.843 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.15-3.el8.x86_64 2022-07-18T03:00:43.843 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-07-18T03:00:43.843 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-07-18T03:00:43.844 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-07-18T03:00:43.844 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-07-18T03:00:43.844 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:43.844 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:43.877 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-07-18T03:00:43.952 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-rook 2022-07-18T03:00:44.100 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-07-18T03:00:44.320 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 5/5 2022-07-18T03:00:44.488 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:52 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:44.571 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 5/5 2022-07-18T03:00:44.571 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/5 2022-07-18T03:00:44.571 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-07-18T03:00:44.572 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-07-18T03:00:44.572 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-07-18T03:00:44.941 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-07-18T03:00:44.942 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:44.947 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:44.948 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:44.948 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-07-18T03:00:44.949 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.5.1-7.el8.noarch 2022-07-18T03:00:44.949 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-07-18T03:00:44.950 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-07-18T03:00:44.950 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:44.951 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:45.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse 2022-07-18T03:00:45.463 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:45.465 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================= 2022-07-18T03:00:45.465 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:45.465 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================= 2022-07-18T03:00:45.466 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:45.466 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 47 k 2022-07-18T03:00:45.466 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T03:00:45.466 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-07-18T03:00:45.466 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-07-18T03:00:45.467 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-07-18T03:00:45.467 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-07-18T03:00:45.467 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-07-18T03:00:45.467 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-07-18T03:00:45.468 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-07-18T03:00:45.468 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-07-18T03:00:45.469 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-07-18T03:00:45.469 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-07-18T03:00:45.469 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-07-18T03:00:45.469 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-07-18T03:00:45.469 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:45.470 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:45.470 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================= 2022-07-18T03:00:45.472 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2022-07-18T03:00:45.472 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:45.474 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.9 M 2022-07-18T03:00:45.474 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 25 M 2022-07-18T03:00:45.475 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:45.592 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:54 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:45.795 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 81 kB/s | 27 kB 00:00 2022-07-18T03:00:45.824 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 57 kB/s | 20 kB 00:00 2022-07-18T03:00:45.862 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.4 MB/s | 126 kB 00:00 2022-07-18T03:00:45.872 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-07-18T03:00:45.900 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2022-07-18T03:00:45.923 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): ceph-mgr-rook-17.0.0-13595.gff10064a.el 103 kB/s | 47 kB 00:00 2022-07-18T03:00:45.956 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.5 MB/s | 155 kB 00:00 2022-07-18T03:00:46.040 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2022-07-18T03:00:46.098 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 171 kB/s | 33 kB 00:00 2022-07-18T03:00:46.115 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 511 kB/s | 81 kB 00:00 2022-07-18T03:00:46.171 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2022-07-18T03:00:46.193 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 612 kB/s | 61 kB 00:00 2022-07-18T03:00:46.268 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.4 MB/s | 1.2 MB 00:00 2022-07-18T03:00:46.268 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:46.269 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.4 MB/s | 1.9 MB 00:00 2022-07-18T03:00:46.321 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:46.334 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:46.335 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:46.429 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:46.429 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:46.552 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:46.552 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:46.552 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:46.552 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:46.553 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:46.553 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:17.0.0-13595.gff10064a.el8 ceph 883 k 2022-07-18T03:00:46.554 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2022-07-18T03:00:46.554 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-07-18T03:00:46.555 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:46.555 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:46.555 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:46.555 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2022-07-18T03:00:46.557 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:46.557 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 965 k 2022-07-18T03:00:46.558 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 3.1 M 2022-07-18T03:00:46.558 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:46.654 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 784 kB/s | 83 kB 00:00 2022-07-18T03:00:46.869 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-fuse-17.0.0-13595.gff10064a.el8.x86 2.7 MB/s | 883 kB 00:00 2022-07-18T03:00:46.870 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:46.870 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.9 MB/s | 965 kB 00:00 2022-07-18T03:00:46.886 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:46.911 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:46.911 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:46.957 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:46.978 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:46.979 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:47.147 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:47.214 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-07-18T03:00:47.408 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-07-18T03:00:47.413 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-07-18T03:00:47.575 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 2/2 2022-07-18T03:00:47.667 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-07-18T03:00:47.861 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-07-18T03:00:47.929 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 2/2 2022-07-18T03:00:47.930 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/2 2022-07-18T03:00:48.025 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-07-18T03:00:48.239 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-07-18T03:00:48.309 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-07-18T03:00:48.309 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:48.310 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:48.310 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-07-18T03:00:48.311 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:48.311 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:48.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-volume 2022-07-18T03:00:48.531 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-07-18T03:00:48.748 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-07-18T03:00:49.037 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:58 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:49.756 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-07-18T03:00:50.005 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-07-18T03:00:50.032 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:50.032 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:50.032 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2022-07-18T03:00:50.033 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:50.033 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:50.070 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 260 k 2022-07-18T03:00:50.070 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:50.071 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:50.071 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:50.076 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:00:50.083 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:50.083 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 260 k 2022-07-18T03:00:50.083 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.3 M 2022-07-18T03:00:50.083 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-07-18T03:00:50.238 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-volume-17.0.0-13595.gff10064a.el8.noarch.r 1.2 MB/s | 260 kB 00:00 2022-07-18T03:00:50.241 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:50.241 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.2 MB/s | 260 kB 00:00 2022-07-18T03:00:50.242 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:50.261 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:50.261 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:50.290 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:50.291 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:50.410 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-07-18T03:00:50.440 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:50.553 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 13/13 2022-07-18T03:00:50.684 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:50.816 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 13/13 2022-07-18T03:00:50.817 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 1/13 2022-07-18T03:00:50.817 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-07-18T03:00:50.817 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-07-18T03:00:50.818 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-07-18T03:00:50.818 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-07-18T03:00:50.819 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-07-18T03:00:50.819 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-07-18T03:00:50.820 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-07-18T03:00:50.820 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-07-18T03:00:50.824 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-07-18T03:00:50.824 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-07-18T03:00:50.825 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-07-18T03:00:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:51.390 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-07-18T03:00:51.391 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:51.391 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:51.391 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:51.391 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-07-18T03:00:51.406 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-07-18T03:00:51.407 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-07-18T03:00:51.407 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-07-18T03:00:51.407 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-07-18T03:00:51.407 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-07-18T03:00:51.408 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-07-18T03:00:51.408 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-07-18T03:00:51.408 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-07-18T03:00:51.408 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-07-18T03:00:51.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.8-1.el8.noarch 2022-07-18T03:00:51.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-07-18T03:00:51.409 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:51.409 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:51.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:00:51.412 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:51.412 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:51.412 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:51.413 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:51.413 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:51.492 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-cephadm 2022-07-18T03:00:51.620 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel 2022-07-18T03:00:52.031 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:00 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:52.194 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:01 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:52.991 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:52.991 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:52.991 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:52.992 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:52.992 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:52.992 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 111 k 2022-07-18T03:00:52.993 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T03:00:52.993 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-07-18T03:00:52.993 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-07-18T03:00:52.993 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-07-18T03:00:52.994 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-07-18T03:00:52.994 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:52.994 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:52.994 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:52.995 INFO:teuthology.orchestra.run.smithi133.stdout:Install 5 Packages 2022-07-18T03:00:52.995 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:52.995 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 5.9 M 2022-07-18T03:00:52.995 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 26 M 2022-07-18T03:00:52.996 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:53.065 INFO:teuthology.orchestra.run.smithi133.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.0 MB/s | 538 kB 00:00 2022-07-18T03:00:53.142 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:53.145 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:53.146 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2022-07-18T03:00:53.146 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:53.147 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:53.147 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:17.0.0-13595.gff10064a.el8 ceph 125 k 2022-07-18T03:00:53.148 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:53.148 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:53.148 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:53.149 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:00:53.149 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:53.149 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 125 k 2022-07-18T03:00:53.149 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 431 k 2022-07-18T03:00:53.150 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2022-07-18T03:00:53.231 INFO:teuthology.orchestra.run.smithi133.stdout:(3/5): ceph-mgr-cephadm-17.0.0-13595.gff10064a. 533 kB/s | 111 kB 00:00 2022-07-18T03:00:53.353 INFO:teuthology.orchestra.run.smithi133.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.6 MB/s | 476 kB 00:00 2022-07-18T03:00:53.358 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-17.0.0-13595.gff10064a.el8.x86_6 583 kB/s | 125 kB 00:00 2022-07-18T03:00:53.360 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:53.361 INFO:teuthology.orchestra.run.smithi159.stdout:Total 580 kB/s | 125 kB 00:00 2022-07-18T03:00:53.361 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:53.372 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:53.373 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:53.424 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:53.438 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:53.466 INFO:teuthology.orchestra.run.smithi133.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 212 kB/s | 60 kB 00:00 2022-07-18T03:00:53.467 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:53.467 INFO:teuthology.orchestra.run.smithi133.stdout:Total 13 MB/s | 5.9 MB 00:00 2022-07-18T03:00:53.507 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:53.531 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:53.531 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:53.573 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:53.599 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:53.611 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:00:53.844 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:53.996 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:00:54.196 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-07-18T03:00:54.399 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:00:54.399 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:54.400 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:54.400 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:54.401 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:54.401 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:54.583 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2 2022-07-18T03:00:55.153 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:04 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:55.153 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-07-18T03:00:55.487 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-07-18T03:00:55.738 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-07-18T03:00:55.938 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 5/5 2022-07-18T03:00:56.055 INFO:teuthology.orchestra.run.smithi159.stdout:Package libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:00:56.123 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:56.124 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:00:56.124 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:56.207 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 5/5 2022-07-18T03:00:56.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 1/5 2022-07-18T03:00:56.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-07-18T03:00:56.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-07-18T03:00:56.242 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-07-18T03:00:56.428 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs-devel 2022-07-18T03:00:56.607 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-07-18T03:00:56.607 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:56.608 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:00:56.608 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:00:56.608 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-07-18T03:00:56.609 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.5.1-7.el8.noarch 2022-07-18T03:00:56.609 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-07-18T03:00:56.610 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-07-18T03:00:56.610 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:56.611 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:00:56.717 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse 2022-07-18T03:00:57.010 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:05 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:57.256 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:05 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:00:57.943 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:00:57.944 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:57.944 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2022-07-18T03:00:57.944 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:57.944 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:00:57.945 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:17.0.0-13595.gff10064a.el8 ceph 30 k 2022-07-18T03:00:57.945 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:57.945 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:00:57.945 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:00:57.945 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:00:57.969 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:57.970 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 30 k 2022-07-18T03:00:57.970 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 136 k 2022-07-18T03:00:57.970 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:00:58.125 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs-devel-17.0.0-13595.gff10064a.el8.x86_ 170 kB/s | 30 kB 00:00 2022-07-18T03:00:58.125 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:58.126 INFO:teuthology.orchestra.run.smithi159.stdout:Total 169 kB/s | 30 kB 00:00 2022-07-18T03:00:58.126 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:00:58.126 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:00:58.126 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:00:58.162 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:00:58.163 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:00:58.249 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:00:58.268 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:00:58.269 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:58.269 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:00:58.269 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:58.273 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:00:58.273 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:17.0.0-13595.gff10064a.el8 ceph 883 k 2022-07-18T03:00:58.274 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2022-07-18T03:00:58.274 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-07-18T03:00:58.275 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:58.275 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:00:58.276 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:00:58.276 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2022-07-18T03:00:58.276 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:00:58.277 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 965 k 2022-07-18T03:00:58.277 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 3.1 M 2022-07-18T03:00:58.278 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:00:58.307 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2022-07-18T03:00:58.431 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:00:58.676 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:00:59.018 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:00:59.018 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:59.019 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:00:59.019 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:00:59.019 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:00:59.019 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:00:59.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2 2022-07-18T03:00:59.326 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-fuse-17.0.0-13595.gff10064a.el8.x86 811 kB/s | 883 kB 00:01 2022-07-18T03:00:59.327 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:00:59.327 INFO:teuthology.orchestra.run.smithi133.stdout:Total 885 kB/s | 965 kB 00:01 2022-07-18T03:00:59.328 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:00:59.347 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:00:59.348 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:00:59.425 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:00:59.426 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:00:59.583 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:00:59.680 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:08 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:00:59.853 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-07-18T03:01:00.059 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 2/2 2022-07-18T03:01:00.438 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 2/2 2022-07-18T03:01:00.439 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/2 2022-07-18T03:01:00.598 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:00.650 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:00.653 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:01:00.653 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:00.735 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1 2022-07-18T03:01:00.775 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-07-18T03:01:00.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:00.776 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:00.776 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-07-18T03:01:00.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:00.777 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:00.959 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-volume 2022-07-18T03:01:01.287 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:10 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:01.522 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:09 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:02.187 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:02.246 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:02.247 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:01:02.248 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:02.495 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:02.495 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:02.496 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2022-07-18T03:01:02.496 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:02.496 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:01:02.496 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume noarch 2:17.0.0-13595.gff10064a.el8 ceph-noarch 260 k 2022-07-18T03:01:02.497 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:02.497 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:01:02.507 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:02.507 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:01:02.508 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:02.508 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 260 k 2022-07-18T03:01:02.508 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.3 M 2022-07-18T03:01:02.508 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:01:02.529 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados 2022-07-18T03:01:02.986 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-volume-17.0.0-13595.gff10064a.el8.noarch.r 516 kB/s | 260 kB 00:00 2022-07-18T03:01:03.008 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:03.008 INFO:teuthology.orchestra.run.smithi133.stdout:Total 515 kB/s | 260 kB 00:00 2022-07-18T03:01:03.009 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:01:03.009 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:01:03.026 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:01:03.028 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:01:03.028 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:01:03.085 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:12 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:03.182 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:01:03.443 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:01:03.809 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:01:03.985 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:04.057 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:04.058 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:01:04.058 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:04.132 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw 2022-07-18T03:01:04.143 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 1/1 2022-07-18T03:01:04.143 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:04.144 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:04.144 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume-2:17.0.0-13595.gff10064a.el8.noarch 2022-07-18T03:01:04.144 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:04.145 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:04.326 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel 2022-07-18T03:01:04.680 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:13 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:04.928 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:12 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:05.596 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:05.658 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:05.659 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:01:05.660 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:05.750 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs 2022-07-18T03:01:05.883 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:05.883 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:05.884 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2022-07-18T03:01:05.884 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:05.885 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:01:05.885 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:17.0.0-13595.gff10064a.el8 ceph 125 k 2022-07-18T03:01:05.886 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:05.886 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:01:05.887 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:05.887 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:01:05.887 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:05.888 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 125 k 2022-07-18T03:01:05.888 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 431 k 2022-07-18T03:01:05.889 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:01:06.125 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-17.0.0-13595.gff10064a.el8.x86_6 520 kB/s | 125 kB 00:00 2022-07-18T03:01:06.126 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:06.126 INFO:teuthology.orchestra.run.smithi133.stdout:Total 518 kB/s | 125 kB 00:00 2022-07-18T03:01:06.127 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:01:06.128 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:01:06.129 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:01:06.184 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:01:06.223 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:01:06.295 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:15 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:06.300 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:01:06.508 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:06.775 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:07.207 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:07.306 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:07.306 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:01:07.307 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:07.344 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd 2022-07-18T03:01:07.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:07.387 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:07.388 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:07.388 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:07.389 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:07.390 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:07.590 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2 2022-07-18T03:01:07.921 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:16 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:08.138 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:16 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:08.812 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:08.875 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:08.876 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2022-07-18T03:01:08.876 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:08.956 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse 2022-07-18T03:01:09.052 INFO:teuthology.orchestra.run.smithi133.stdout:Package libcephfs2-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:09.116 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:09.117 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:09.117 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:09.191 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs-devel 2022-07-18T03:01:09.510 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:18 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:09.728 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:17 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:10.461 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:10.466 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:10.466 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2022-07-18T03:01:10.466 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:10.466 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:01:10.467 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:17.0.0-13595.gff10064a.el8 ceph 94 k 2022-07-18T03:01:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:10.483 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:01:10.487 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:10.488 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:01:10.488 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:10.489 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 94 k 2022-07-18T03:01:10.511 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 284 k 2022-07-18T03:01:10.511 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:01:10.671 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-17.0.0-13595.gff10064a.el8.x86_64.rpm 468 kB/s | 94 kB 00:00 2022-07-18T03:01:10.671 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:10.672 INFO:teuthology.orchestra.run.smithi159.stdout:Total 465 kB/s | 94 kB 00:00 2022-07-18T03:01:10.672 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:01:10.673 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:01:10.682 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:01:10.710 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:10.710 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:10.711 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2022-07-18T03:01:10.711 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:10.711 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:01:10.711 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:17.0.0-13595.gff10064a.el8 ceph 30 k 2022-07-18T03:01:10.712 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:10.712 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:01:10.712 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:10.712 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:01:10.712 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:10.713 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 30 k 2022-07-18T03:01:10.713 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 136 k 2022-07-18T03:01:10.714 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:01:10.720 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:01:10.723 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:01:10.831 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:01:10.849 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-devel-17.0.0-13595.gff10064a.el8.x86_ 212 kB/s | 30 kB 00:00 2022-07-18T03:01:10.856 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:10.856 INFO:teuthology.orchestra.run.smithi133.stdout:Total 210 kB/s | 30 kB 00:00 2022-07-18T03:01:10.857 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:01:10.857 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:01:10.857 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:01:10.890 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:01:10.891 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:01:10.949 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:01:11.055 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:11.123 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:11.305 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:11.347 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:11.683 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:11.684 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:11.684 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:11.684 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:11.685 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:11.685 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:11.712 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:11.713 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:11.713 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:01:11.713 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:11.713 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:11.713 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:11.767 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2 2022-07-18T03:01:11.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror 2022-07-18T03:01:12.308 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:20 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:12.439 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:21 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:13.213 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:13.273 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:13.274 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:13.278 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:13.416 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:13.418 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:13.419 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2022-07-18T03:01:13.419 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:13.419 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:01:13.420 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.8 M 2022-07-18T03:01:13.420 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 3.8 M 2022-07-18T03:01:13.424 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 19 M 2022-07-18T03:01:13.424 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:01:14.049 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1 2022-07-18T03:01:14.163 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-17.0.0-13595.gff10064a.el8.x86_64.rp 5.0 MB/s | 3.8 MB 00:00 2022-07-18T03:01:14.164 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:14.164 INFO:teuthology.orchestra.run.smithi159.stdout:Total 5.0 MB/s | 3.8 MB 00:00 2022-07-18T03:01:14.164 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:01:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:01:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:01:14.235 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:01:14.241 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:01:14.596 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:22 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:14.862 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:01:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:15.435 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:15.525 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:15.578 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:15.579 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:15.579 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:15.644 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados 2022-07-18T03:01:15.809 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:15.810 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:15.810 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:01:15.811 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:15.811 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:15.812 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:16.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd 2022-07-18T03:01:16.217 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:24 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:16.590 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:25 ago on Mon 18 Jul 2022 02:58:51 AM UTC. 2022-07-18T03:01:17.121 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rados-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:17.186 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:17.186 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:17.187 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:17.261 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw 2022-07-18T03:01:17.554 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2022-07-18T03:01:17.554 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:17.554 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2022-07-18T03:01:17.554 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:17.555 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2022-07-18T03:01:17.556 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:17.0.0-13595.gff10064a.el8 ceph 176 k 2022-07-18T03:01:17.556 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:17.557 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2022-07-18T03:01:17.557 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2022-07-18T03:01:17.557 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2022-07-18T03:01:17.558 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:17.559 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 176 k 2022-07-18T03:01:17.559 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 567 k 2022-07-18T03:01:17.559 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2022-07-18T03:01:17.755 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-17.0.0-13595.gff10064a.el8.x86_64.rpm 885 kB/s | 176 kB 00:00 2022-07-18T03:01:17.755 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:17.756 INFO:teuthology.orchestra.run.smithi159.stdout:Total 880 kB/s | 176 kB 00:00 2022-07-18T03:01:17.756 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2022-07-18T03:01:17.789 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2022-07-18T03:01:17.789 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2022-07-18T03:01:17.813 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:25 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:17.816 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2022-07-18T03:01:17.817 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2022-07-18T03:01:17.940 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2022-07-18T03:01:18.123 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:18.411 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:18.718 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rgw-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:18.773 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:18.774 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:18.774 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2022-07-18T03:01:18.775 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:18.775 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:18.775 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2022-07-18T03:01:18.787 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:18.787 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:18.787 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:18.850 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs 2022-07-18T03:01:18.941 DEBUG:teuthology.parallel:result is None 2022-07-18T03:01:19.394 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:27 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:20.309 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-cephfs-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:20.390 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:20.393 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:20.393 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:20.446 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd 2022-07-18T03:01:21.011 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:28 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:21.902 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rbd-2:17.0.0-13595.gff10064a.el8.x86_64 is already installed. 2022-07-18T03:01:21.967 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:21.968 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2022-07-18T03:01:21.968 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:22.035 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse 2022-07-18T03:01:22.576 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:30 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:23.539 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:23.540 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:23.540 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2022-07-18T03:01:23.540 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:23.541 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:01:23.541 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:17.0.0-13595.gff10064a.el8 ceph 94 k 2022-07-18T03:01:23.545 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:23.546 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:01:23.547 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:23.548 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:01:23.548 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:23.549 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 94 k 2022-07-18T03:01:23.550 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 284 k 2022-07-18T03:01:23.551 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:01:23.907 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-17.0.0-13595.gff10064a.el8.x86_64.rpm 260 kB/s | 94 kB 00:00 2022-07-18T03:01:23.908 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:23.908 INFO:teuthology.orchestra.run.smithi133.stdout:Total 260 kB/s | 94 kB 00:00 2022-07-18T03:01:23.909 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:01:23.909 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:01:23.909 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:01:23.950 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:01:23.958 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:01:24.061 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:01:24.262 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:24.517 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:24.899 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:24.900 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:24.900 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:24.901 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:24.901 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:24.902 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:25.068 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror 2022-07-18T03:01:25.659 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:33 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:26.585 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:26.587 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:26.597 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2022-07-18T03:01:26.597 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:26.602 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:01:26.602 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:17.0.0-13595.gff10064a.el8 ceph 3.8 M 2022-07-18T03:01:26.603 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:26.603 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:01:26.604 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:26.604 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:01:26.618 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:26.619 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 3.8 M 2022-07-18T03:01:26.619 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 19 M 2022-07-18T03:01:26.620 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:01:28.458 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-17.0.0-13595.gff10064a.el8.x86_64.rp 2.0 MB/s | 3.8 MB 00:01 2022-07-18T03:01:28.459 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:28.459 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.0 MB/s | 3.8 MB 00:01 2022-07-18T03:01:28.459 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:01:28.473 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:01:28.473 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:01:28.548 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:01:28.555 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:01:29.024 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:01:29.181 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:29.550 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:29.914 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:29.915 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:29.915 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:29.916 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:29.916 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:29.916 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:30.097 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd 2022-07-18T03:01:30.657 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:38 ago on Mon 18 Jul 2022 02:58:52 AM UTC. 2022-07-18T03:01:31.583 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2022-07-18T03:01:31.585 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:31.586 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2022-07-18T03:01:31.586 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:31.586 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2022-07-18T03:01:31.587 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:17.0.0-13595.gff10064a.el8 ceph 176 k 2022-07-18T03:01:31.587 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:31.588 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2022-07-18T03:01:31.588 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2022-07-18T03:01:31.589 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2022-07-18T03:01:31.589 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:31.602 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 176 k 2022-07-18T03:01:31.602 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 567 k 2022-07-18T03:01:31.602 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2022-07-18T03:01:31.841 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-17.0.0-13595.gff10064a.el8.x86_64.rpm 691 kB/s | 176 kB 00:00 2022-07-18T03:01:31.842 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2022-07-18T03:01:31.844 INFO:teuthology.orchestra.run.smithi133.stdout:Total 688 kB/s | 176 kB 00:00 2022-07-18T03:01:31.845 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2022-07-18T03:01:31.852 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2022-07-18T03:01:31.852 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2022-07-18T03:01:31.915 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2022-07-18T03:01:31.916 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2022-07-18T03:01:32.031 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2022-07-18T03:01:32.169 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:32.414 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:32.795 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 1/1 2022-07-18T03:01:32.796 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:32.796 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2022-07-18T03:01:32.797 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:17.0.0-13595.gff10064a.el8.x86_64 2022-07-18T03:01:32.798 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:32.798 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2022-07-18T03:01:33.006 DEBUG:teuthology.parallel:result is None 2022-07-18T03:01:33.008 DEBUG:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2022-07-18T03:01:33.050 INFO:teuthology.orchestra.run.smithi133.stdout:CentOSStream 2022-07-18T03:01:33.053 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-07-18T03:01:33.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ff10064ab878e58afb1d2538b2086a2d4a41c017 2022-07-18T03:01:33.241 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-07-18T03:01:33.266 INFO:teuthology.orchestra.run.smithi133.stdout:17.0.0-13595.gff10064a.el8 2022-07-18T03:01:33.268 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13595.gff10064a.el8 2022-07-18T03:01:33.268 INFO:teuthology.task.install:The correct ceph version 17.0.0-13595.gff10064a is installed. 2022-07-18T03:01:33.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo lsb_release -is 2022-07-18T03:01:33.315 INFO:teuthology.orchestra.run.smithi159.stdout:CentOSStream 2022-07-18T03:01:33.317 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-07-18T03:01:33.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ff10064ab878e58afb1d2538b2086a2d4a41c017 2022-07-18T03:01:33.480 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-07-18T03:01:33.518 INFO:teuthology.orchestra.run.smithi159.stdout:17.0.0-13595.gff10064a.el8 2022-07-18T03:01:33.519 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13595.gff10064a.el8 2022-07-18T03:01:33.519 INFO:teuthology.task.install:The correct ceph version 17.0.0-13595.gff10064a is installed. 2022-07-18T03:01:33.520 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-07-18T03:01:33.520 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:33.521 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-18T03:01:33.566 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:01:33.574 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-18T03:01:33.627 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-07-18T03:01:33.628 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:33.628 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2022-07-18T03:01:33.658 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-18T03:01:33.727 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:01:33.728 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2022-07-18T03:01:33.765 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-18T03:01:33.874 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-07-18T03:01:33.875 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:33.876 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-18T03:01:33.931 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-18T03:01:34.004 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:01:34.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-18T03:01:34.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-18T03:01:34.113 INFO:teuthology.run_tasks:Running task ceph... 2022-07-18T03:01:34.222 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-07-18T03:01:34.224 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2022-07-18T03:01:34.231 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2022-07-18T03:01:34.264 INFO:tasks.ceph:Disabling ceph logrotate... 2022-07-18T03:01:34.264 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-18T03:01:34.297 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-18T03:01:34.357 INFO:tasks.ceph:Creating extra log directories... 2022-07-18T03:01:34.358 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-18T03:01:34.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-18T03:01:34.495 INFO:tasks.ceph:Creating ceph cluster c1... 2022-07-18T03:01:34.496 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-07-18T03:01:34.497 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300', 'branch': 'wip-rgwlc-azone', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6935300', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-rgwlc-azone'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-rgwlc-azone', 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017'}}, 'owner': 'scheduled_mbenjamin@shatner', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgwlc-azone', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'targets': {'smithi133.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbz643aJzTA1MGouCBCaX2wAlr7gxhsuzbiWpPfB1J+XwHbAt+kDOGp2Ft/jOzaXFKrq9e20ENTcd5lzC730MsuhxkXAFpXApXKyIfdGdowsKOGictcs22h7dXfvGGl7lytZg65AHT5kAGQZkADb24rTTvlM9ZtQrM2wLMo5pHBM4p+2nlHFEMH4NvWM0W4qQ+MUTv+V9EbLfB3Kf4OJcfMuw4S9+fSDf3bqVb8I7WIx/wtYFxEEJT4PolE/dvR7kCe/C6rqqqDMP154xs5/17c652VftNZq41Vv8IueKmd3wq2ZOM2zqrNLCGr1FkE7T4jBC89zWOCn9PRu+RF9Vl0lPLKOrkVOVOoQdEZSZixPiOEkRyKEkWNxkyIPtS/EDTjjDKIZSrCeRLm3EOXMP0eFTEa98qWrsoaxtLOeDIPM3HwufNX5YU5DLyvmx+x9ZQilp3QwJD89g+dzQqSSdDgxjTvJ2ZDvid4MkWd8G5Faw/UCUtHNq8FLc9BCGZ+pM=', 'smithi159.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0XX/4wSzWw+RBIBAqfnofJBXzIjtevgLHZtgd1wszCkZ2NrfTAt7OZWuj/Imft/uz6GkoqE0v5k/VRYEThvU4Fg0FCIKaBPOc7rZOR3Ego4Qk+Xun+Fm0AfDMCVtZTsCY/dImi9Kh2bOruLyGZnRPD4hnLoYNgsHioSMJjGOyWj5cIzGU7u/UNmmfSDvQqTnBUbvXYWLuRkwYDnyUGt36BVOM+OHhS3ODwnKXtNycReRYuxdoh8T4Q193sdYXaaISZxtQnPiIG/tRMy7vDanx5m+m9I0i65kh/J4umQQuqQ/DutpXO8TrPXAxqYtkyloh9IrBRV1bwN55ziIA9cPLHl9LBFA7cGKjk/2Bm5tEz9DxeMWUPygu6jFc+UUh3zNTaaJp9nSPMYmwYP8Ub7TmEluVDqPJh6jctHSYllsA/scLg19rIGTi7E1s7KxLOu8bBT++U9T6HPeVXqwZqZlLmEoIBKTE2wnIS5NHj/M/pdzfAvd/OqdUsOJGxMTJHDc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'args': ['-a', '!bucket_sync_disable,!bucket_reshard'], 'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'main', 'teuthology_sha1': '89d874d555f56b1c1f383df72487aecec0ee6cf9', 'timestamp': '2022-07-17_22:24:15', 'tube': 'smithi', 'user': 'mbenjamin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-07-18T03:01:34.507 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-07-18T03:01:34.517 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-07-18T03:01:34.548 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2022-07-18T03:01:34.590 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2022-07-18T03:01:34.640 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:34.640 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2022-07-18T03:01:34.676 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-18T03:01:34.677 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-18T03:01:34.762 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-18T03:01:34.763 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:01:34.764 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 123669 Links: 1 2022-07-18T03:01:34.764 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:01:34.765 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:01:34.765 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2022-07-18 03:01:29.334334084 +0000 2022-07-18T03:01:34.766 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2022-07-18 02:57:54.610755192 +0000 2022-07-18T03:01:34.766 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2022-07-18 02:57:54.610755192 +0000 2022-07-18T03:01:34.766 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2022-07-18T03:01:34.772 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2022-07-18T03:01:34.772 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2022-07-18T03:01:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000234878 s, 2.2 MB/s 2022-07-18T03:01:34.784 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-18T03:01:34.834 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-18T03:01:34.835 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:01:34.835 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 124077 Links: 1 2022-07-18T03:01:34.835 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:01:34.836 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:01:34.836 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2022-07-18 03:01:29.334334084 +0000 2022-07-18T03:01:34.836 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2022-07-18 02:57:55.324742198 +0000 2022-07-18T03:01:34.836 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2022-07-18 02:57:55.324742198 +0000 2022-07-18T03:01:34.836 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2022-07-18T03:01:34.847 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2022-07-18T03:01:34.848 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2022-07-18T03:01:34.848 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000219462 s, 2.3 MB/s 2022-07-18T03:01:34.849 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-18T03:01:34.907 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-18T03:01:34.907 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:01:34.908 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 126253 Links: 1 2022-07-18T03:01:34.908 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:01:34.909 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:01:34.909 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2022-07-18 03:01:29.334334084 +0000 2022-07-18T03:01:34.910 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2022-07-18 02:57:56.010729714 +0000 2022-07-18T03:01:34.910 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2022-07-18 02:57:56.010729714 +0000 2022-07-18T03:01:34.912 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2022-07-18T03:01:34.914 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2022-07-18T03:01:34.914 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2022-07-18T03:01:34.915 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000216301 s, 2.4 MB/s 2022-07-18T03:01:34.919 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-18T03:01:34.991 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-18T03:01:34.992 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:01:34.992 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 126363 Links: 1 2022-07-18T03:01:34.993 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:01:34.993 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:01:35.028 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2022-07-18 03:01:29.334334084 +0000 2022-07-18T03:01:35.029 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2022-07-18 02:57:56.622718576 +0000 2022-07-18T03:01:35.029 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2022-07-18 02:57:56.622718576 +0000 2022-07-18T03:01:35.045 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2022-07-18T03:01:35.047 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2022-07-18T03:01:35.048 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2022-07-18T03:01:35.048 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000215733 s, 2.4 MB/s 2022-07-18T03:01:35.049 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:01:35.049 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-07-18T03:01:35.085 INFO:tasks.ceph:Generating config... 2022-07-18T03:01:35.103 INFO:tasks.ceph:[client] debug rgw = 20 2022-07-18T03:01:35.104 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-07-18T03:01:35.105 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-07-18T03:01:35.105 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-07-18T03:01:35.114 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-07-18T03:01:35.114 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-07-18T03:01:35.115 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-07-18T03:01:35.115 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-07-18T03:01:35.116 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-07-18T03:01:35.117 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-07-18T03:01:35.162 INFO:tasks.ceph:[client] setgroup = ceph 2022-07-18T03:01:35.163 INFO:tasks.ceph:[client] setuser = ceph 2022-07-18T03:01:35.163 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-07-18T03:01:35.163 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-07-18T03:01:35.164 INFO:tasks.ceph:[mgr] debug ms = 1 2022-07-18T03:01:35.164 INFO:tasks.ceph:[mon] debug mon = 20 2022-07-18T03:01:35.164 INFO:tasks.ceph:[mon] debug ms = 1 2022-07-18T03:01:35.165 INFO:tasks.ceph:[mon] debug paxos = 20 2022-07-18T03:01:35.165 INFO:tasks.ceph:[osd] debug ms = 1 2022-07-18T03:01:35.165 INFO:tasks.ceph:[osd] debug osd = 20 2022-07-18T03:01:35.166 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-07-18T03:01:35.166 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-07-18T03:01:35.166 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-07-18T03:01:35.167 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-07-18T03:01:35.167 INFO:tasks.ceph:Setting up c1.mon.a... 2022-07-18T03:01:35.168 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2022-07-18T03:01:35.235 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c1.keyring 2022-07-18T03:01:35.235 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2022-07-18T03:01:35.281 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-07-18T03:01:35.364 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.133')] 2022-07-18T03:01:35.387 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.133'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-07-18T03:01:35.400 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:35.400 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-07-18T03:01:35.459 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.133 --print /home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:35.539 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:01:35.540 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:01:35.544 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:35.544 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: generated fsid c640de7a-c5e8-4a81-b782-87c1207bd4be 2022-07-18T03:01:35.545 INFO:teuthology.orchestra.run.smithi133.stdout:setting min_mon_release = octopus 2022-07-18T03:01:35.545 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 0 2022-07-18T03:01:35.545 INFO:teuthology.orchestra.run.smithi133.stdout:fsid c640de7a-c5e8-4a81-b782-87c1207bd4be 2022-07-18T03:01:35.554 INFO:teuthology.orchestra.run.smithi133.stdout:last_changed 2022-07-18T03:01:35.538122+0000 2022-07-18T03:01:35.555 INFO:teuthology.orchestra.run.smithi133.stdout:created 2022-07-18T03:01:35.538122+0000 2022-07-18T03:01:35.555 INFO:teuthology.orchestra.run.smithi133.stdout:min_mon_release 15 (octopus) 2022-07-18T03:01:35.556 INFO:teuthology.orchestra.run.smithi133.stdout:election_strategy: 1 2022-07-18T03:01:35.556 INFO:teuthology.orchestra.run.smithi133.stdout:0: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.a 2022-07-18T03:01:35.556 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2022-07-18T03:01:35.558 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-07-18T03:01:35.641 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID c640de7a-c5e8-4a81-b782-87c1207bd4be... 2022-07-18T03:01:35.643 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-07-18T03:01:35.689 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-07-18T03:01:35.746 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2022-07-18T03:01:35.748 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2022-07-18T03:01:35.749 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-18T03:01:35.749 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2022-07-18T03:01:35.749 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.774 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2022-07-18T03:01:35.775 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.775 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2022-07-18T03:01:35.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.776 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2022-07-18T03:01:35.777 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2022-07-18T03:01:35.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.778 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2022-07-18T03:01:35.785 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2022-07-18T03:01:35.786 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.787 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-18T03:01:35.790 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2022-07-18T03:01:35.802 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.803 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2022-07-18T03:01:35.803 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-18T03:01:35.803 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.804 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2022-07-18T03:01:35.804 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.805 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2022-07-18T03:01:35.805 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2022-07-18T03:01:35.806 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2022-07-18T03:01:35.810 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2022-07-18T03:01:35.811 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2022-07-18T03:01:35.811 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2022-07-18T03:01:35.812 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2022-07-18T03:01:35.818 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-18T03:01:35.819 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-18T03:01:35.820 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2022-07-18T03:01:35.820 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.820 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-07-18T03:01:35.821 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.821 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2022-07-18T03:01:35.822 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.822 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2022-07-18T03:01:35.822 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.851 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2022-07-18T03:01:35.852 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2022-07-18T03:01:35.852 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2022-07-18T03:01:35.852 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.133 2022-07-18T03:01:35.853 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = c640de7a-c5e8-4a81-b782-87c1207bd4be 2022-07-18T03:01:35.853 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.853 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2022-07-18T03:01:35.853 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2022-07-18T03:01:35.880 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.880 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2022-07-18T03:01:35.880 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2022-07-18T03:01:35.881 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.881 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2022-07-18T03:01:35.881 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2022-07-18T03:01:35.881 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.882 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2022-07-18T03:01:35.882 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2022-07-18T03:01:35.882 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2022-07-18T03:01:35.882 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.883 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2022-07-18T03:01:35.883 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2022-07-18T03:01:35.883 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.883 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2022-07-18T03:01:35.883 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.884 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2022-07-18T03:01:35.884 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.884 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2022-07-18T03:01:35.884 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.885 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2022-07-18T03:01:35.885 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2022-07-18T03:01:35.885 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.885 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2022-07-18T03:01:35.885 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2022-07-18T03:01:35.923 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2022-07-18T03:01:35.924 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 20 2022-07-18T03:01:35.924 INFO:teuthology.orchestra.run.smithi133.stdout: rocksdb delete range threshold = 0 2022-07-18T03:01:35.924 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.924 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2022-07-18T03:01:35.925 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2022-07-18T03:01:35.925 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2022-07-18T03:01:35.925 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2022-07-18T03:01:35.925 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2022-07-18T03:01:35.956 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:01:35.957 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:01:35.957 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2022-07-18T03:01:35.957 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.957 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2022-07-18T03:01:35.958 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2022-07-18T03:01:35.958 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2022-07-18T03:01:35.958 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2022-07-18T03:01:35.958 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2022-07-18T03:01:35.959 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2022-07-18T03:01:35.959 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2022-07-18T03:01:35.959 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:01:35.959 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2022-07-18T03:01:35.960 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2022-07-18T03:01:35.960 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:01:35.960 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.960 INFO:teuthology.orchestra.run.smithi133.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-18T03:01:35.960 INFO:teuthology.orchestra.run.smithi133.stdout: auth mon ticket ttl = 660 # 11m 2022-07-18T03:01:35.961 INFO:teuthology.orchestra.run.smithi133.stdout: auth service ticket ttl = 240 # 4m 2022-07-18T03:01:35.961 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.961 INFO:teuthology.orchestra.run.smithi133.stdout: # don't complain about insecure global_id in the test suite 2022-07-18T03:01:35.974 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-18T03:01:35.974 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-18T03:01:35.975 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.975 INFO:teuthology.orchestra.run.smithi133.stdout: # 1m isn't quite enough 2022-07-18T03:01:35.976 INFO:teuthology.orchestra.run.smithi133.stdout: mon_down_mkfs_grace = 2m 2022-07-18T03:01:35.977 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.977 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_filestore_osds = false 2022-07-18T03:01:35.985 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:35.986 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2022-07-18T03:01:35.986 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2022-07-18T03:01:35.987 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2022-07-18T03:01:35.987 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2022-07-18T03:01:35.988 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-18T03:01:35.988 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-18T03:01:35.989 INFO:teuthology.orchestra.run.smithi133.stdout: debug rgw = 20 2022-07-18T03:01:35.989 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt require ssl = False 2022-07-18T03:01:36.001 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms backend = testing 2022-07-18T03:01:36.002 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-07-18T03:01:36.003 INFO:teuthology.orchestra.run.smithi133.stdout: rgw curl low speed time = 300 2022-07-18T03:01:36.003 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data log num shards = 4 2022-07-18T03:01:36.003 INFO:teuthology.orchestra.run.smithi133.stdout: rgw md log max shards = 4 2022-07-18T03:01:36.004 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync log trim interval = 0 2022-07-18T03:01:36.011 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync meta inject err probability = 0.1 2022-07-18T03:01:36.012 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync obj etag verify = True 2022-07-18T03:01:36.012 INFO:teuthology.orchestra.run.smithi133.stdout: setgroup = ceph 2022-07-18T03:01:36.013 INFO:teuthology.orchestra.run.smithi133.stdout: setuser = ceph 2022-07-18T03:01:36.014 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2022-07-18T03:01:36.014 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2022-07-18T03:01:36.039 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data notify interval msec = 0 2022-07-18T03:01:36.039 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2022-07-18T03:01:36.040 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10 2022-07-18T03:01:36.040 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2022-07-18T03:01:36.041 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 1000 2022-07-18T03:01:36.041 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2022-07-18T03:01:36.042 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10000 2022-07-18T03:01:36.052 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2022-07-18T03:01:36.053 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2022-07-18T03:01:36.053 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-18T03:01:36.054 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2022-07-18T03:01:36.063 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.063 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2022-07-18T03:01:36.064 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.064 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2022-07-18T03:01:36.065 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.066 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2022-07-18T03:01:36.066 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2022-07-18T03:01:36.066 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.067 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2022-07-18T03:01:36.067 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2022-07-18T03:01:36.068 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.068 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-18T03:01:36.069 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2022-07-18T03:01:36.069 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.069 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2022-07-18T03:01:36.070 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-18T03:01:36.070 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.071 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2022-07-18T03:01:36.085 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.086 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2022-07-18T03:01:36.088 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2022-07-18T03:01:36.088 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2022-07-18T03:01:36.089 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2022-07-18T03:01:36.089 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2022-07-18T03:01:36.090 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2022-07-18T03:01:36.090 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2022-07-18T03:01:36.091 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-18T03:01:36.091 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-18T03:01:36.091 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2022-07-18T03:01:36.092 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.096 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-07-18T03:01:36.096 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.100 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2022-07-18T03:01:36.100 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.100 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2022-07-18T03:01:36.101 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.101 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2022-07-18T03:01:36.102 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2022-07-18T03:01:36.102 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2022-07-18T03:01:36.103 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = 172.21.15.133 2022-07-18T03:01:36.103 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = c640de7a-c5e8-4a81-b782-87c1207bd4be 2022-07-18T03:01:36.103 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.104 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2022-07-18T03:01:36.104 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2022-07-18T03:01:36.105 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.105 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2022-07-18T03:01:36.106 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2022-07-18T03:01:36.106 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.107 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2022-07-18T03:01:36.107 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2022-07-18T03:01:36.108 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.119 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2022-07-18T03:01:36.120 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2022-07-18T03:01:36.120 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2022-07-18T03:01:36.121 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.121 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2022-07-18T03:01:36.177 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2022-07-18T03:01:36.177 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.198 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2022-07-18T03:01:36.199 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.199 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2022-07-18T03:01:36.226 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.227 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2022-07-18T03:01:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2022-07-18T03:01:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2022-07-18T03:01:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2022-07-18T03:01:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2022-07-18T03:01:36.230 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-18T03:01:36.230 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2022-07-18T03:01:36.230 INFO:teuthology.orchestra.run.smithi159.stdout: rocksdb delete range threshold = 0 2022-07-18T03:01:36.230 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.231 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2022-07-18T03:01:36.231 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-18T03:01:36.231 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2022-07-18T03:01:36.231 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-18T03:01:36.231 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-18T03:01:36.233 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:01:36.234 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:01:36.234 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2022-07-18T03:01:36.234 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.234 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2022-07-18T03:01:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-18T03:01:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-18T03:01:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2022-07-18T03:01:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-18T03:01:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2022-07-18T03:01:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2022-07-18T03:01:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:01:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2022-07-18T03:01:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2022-07-18T03:01:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:01:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-18T03:01:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2022-07-18T03:01:36.238 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2022-07-18T03:01:36.238 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.238 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2022-07-18T03:01:36.239 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-18T03:01:36.239 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-18T03:01:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2022-07-18T03:01:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2022-07-18T03:01:36.241 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.241 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2022-07-18T03:01:36.242 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:01:36.242 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2022-07-18T03:01:36.242 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2022-07-18T03:01:36.243 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2022-07-18T03:01:36.251 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2022-07-18T03:01:36.252 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-18T03:01:36.252 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-18T03:01:36.252 INFO:teuthology.orchestra.run.smithi159.stdout: debug rgw = 20 2022-07-18T03:01:36.253 INFO:teuthology.orchestra.run.smithi159.stdout: rgw crypt require ssl = False 2022-07-18T03:01:36.253 INFO:teuthology.orchestra.run.smithi159.stdout: rgw crypt s3 kms backend = testing 2022-07-18T03:01:36.254 INFO:teuthology.orchestra.run.smithi159.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-07-18T03:01:36.261 INFO:teuthology.orchestra.run.smithi159.stdout: rgw curl low speed time = 300 2022-07-18T03:01:36.261 INFO:teuthology.orchestra.run.smithi159.stdout: rgw data log num shards = 4 2022-07-18T03:01:36.262 INFO:teuthology.orchestra.run.smithi159.stdout: rgw md log max shards = 4 2022-07-18T03:01:36.262 INFO:teuthology.orchestra.run.smithi159.stdout: rgw sync log trim interval = 0 2022-07-18T03:01:36.263 INFO:teuthology.orchestra.run.smithi159.stdout: rgw sync meta inject err probability = 0.1 2022-07-18T03:01:36.263 INFO:teuthology.orchestra.run.smithi159.stdout: rgw sync obj etag verify = True 2022-07-18T03:01:36.264 INFO:teuthology.orchestra.run.smithi159.stdout: setgroup = ceph 2022-07-18T03:01:36.264 INFO:teuthology.orchestra.run.smithi159.stdout: setuser = ceph 2022-07-18T03:01:36.265 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2022-07-18T03:01:36.265 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2022-07-18T03:01:36.267 INFO:teuthology.orchestra.run.smithi159.stdout: rgw data notify interval msec = 0 2022-07-18T03:01:36.267 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2022-07-18T03:01:36.268 INFO:teuthology.orchestra.run.smithi159.stdout: osd_max_omap_entries_per_request = 10 2022-07-18T03:01:36.268 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2022-07-18T03:01:36.269 INFO:teuthology.orchestra.run.smithi159.stdout: osd_max_omap_entries_per_request = 1000 2022-07-18T03:01:36.269 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2022-07-18T03:01:36.270 INFO:teuthology.orchestra.run.smithi159.stdout: osd_max_omap_entries_per_request = 10000 2022-07-18T03:01:36.270 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2022-07-18T03:01:36.271 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2022-07-18T03:01:36.337 INFO:tasks.ceph:Copying monmap to all nodes... 2022-07-18T03:01:36.346 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:36.347 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2022-07-18T03:01:36.387 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:36.388 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2022-07-18T03:01:36.683 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2022-07-18T03:01:36.683 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:36.684 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/c1.keyring 2022-07-18T03:01:36.684 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-07-18T03:01:36.773 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:36.774 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:36.986 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2022-07-18T03:01:36.986 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:01:36.987 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/c1.keyring 2022-07-18T03:01:36.987 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-07-18T03:01:37.049 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:01:37.049 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:37.110 INFO:tasks.ceph:Setting up mon nodes... 2022-07-18T03:01:37.111 INFO:tasks.ceph:Setting up mgr nodes... 2022-07-18T03:01:37.112 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2022-07-18T03:01:37.165 INFO:teuthology.orchestra.run.smithi133.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2022-07-18T03:01:37.167 INFO:tasks.ceph:Setting up mds nodes... 2022-07-18T03:01:37.168 INFO:tasks.ceph_client:Setting up client nodes... 2022-07-18T03:01:37.169 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2022-07-18T03:01:37.223 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c1.client.0.keyring 2022-07-18T03:01:37.232 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2022-07-18T03:01:37.316 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c1.client.1.keyring 2022-07-18T03:01:37.330 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-07-18T03:01:37.332 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-07-18T03:01:37.333 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2022-07-18T03:01:37.399 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:01:37.400 INFO:tasks.ceph:role: c1.osd.0 2022-07-18T03:01:37.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2022-07-18T03:01:37.401 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-18T03:01:37.479 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-18T03:01:37.480 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-18T03:01:37.480 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-18T03:01:37.480 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-07-18T03:01:37.481 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-18T03:01:37.481 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2022-07-18T03:01:37.481 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-18T03:01:37.481 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-18T03:01:37.482 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-18T03:01:37.482 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-18T03:01:37.919 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2022-07-18T03:01:37.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2022-07-18T03:01:37.920 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2022-07-18T03:01:38.107 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2022-07-18T03:01:38.148 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2022-07-18T03:01:38.246 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:01:38.247 INFO:tasks.ceph:role: c1.osd.1 2022-07-18T03:01:38.250 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2022-07-18T03:01:38.251 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-18T03:01:38.332 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-18T03:01:38.333 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-18T03:01:38.333 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-18T03:01:38.333 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-07-18T03:01:38.334 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-18T03:01:38.334 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2022-07-18T03:01:38.335 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-18T03:01:38.335 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-18T03:01:38.336 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-18T03:01:38.336 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-18T03:01:38.801 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2022-07-18T03:01:38.802 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2022-07-18T03:01:38.803 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2022-07-18T03:01:38.908 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2022-07-18T03:01:38.963 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2022-07-18T03:01:39.078 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:01:39.079 INFO:tasks.ceph:role: c1.osd.2 2022-07-18T03:01:39.079 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com 2022-07-18T03:01:39.080 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-18T03:01:39.165 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-18T03:01:39.165 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-18T03:01:39.166 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-18T03:01:39.166 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-07-18T03:01:39.167 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-18T03:01:39.167 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2022-07-18T03:01:39.168 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-18T03:01:39.168 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-18T03:01:39.168 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-18T03:01:39.169 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-18T03:01:39.609 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2022-07-18T03:01:39.610 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2022-07-18T03:01:39.618 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2022-07-18T03:01:39.806 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2022-07-18T03:01:39.855 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:39.977 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:39.958+0000 7f11b96e5440 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2022-07-18T03:01:39.979 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:39.960+0000 7f11b96e5440 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2022-07-18T03:01:39.979 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:39.960+0000 7f11b96e5440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-07-18T03:01:39.980 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:39.960+0000 7f11b96e5440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-07-18T03:01:39.980 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:39.960+0000 7f11b96e5440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-07-18T03:01:39.981 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:39.961+0000 7f11b96e5440 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2022-07-18T03:01:42.082 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2022-07-18T03:01:42.145 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:42.303 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:42.244+0000 7f71663c4440 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2022-07-18T03:01:42.303 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:42.245+0000 7f71663c4440 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2022-07-18T03:01:42.304 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:42.245+0000 7f71663c4440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-07-18T03:01:42.304 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:42.245+0000 7f71663c4440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-07-18T03:01:42.305 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:42.245+0000 7f71663c4440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-07-18T03:01:42.305 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:42.245+0000 7f71663c4440 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2022-07-18T03:01:44.320 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2022-07-18T03:01:44.391 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:44.504 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:44.501+0000 7f7b2bacb440 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2022-07-18T03:01:44.505 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:44.502+0000 7f7b2bacb440 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2022-07-18T03:01:44.514 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:44.502+0000 7f7b2bacb440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-07-18T03:01:44.515 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:44.502+0000 7f7b2bacb440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-07-18T03:01:44.515 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:44.502+0000 7f7b2bacb440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-07-18T03:01:44.516 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:01:44.502+0000 7f7b2bacb440 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2022-07-18T03:01:46.583 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2022-07-18T03:01:46.675 INFO:tasks.ceph:Reading keys from all nodes... 2022-07-18T03:01:46.676 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:46.676 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2022-07-18T03:01:46.759 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:46.760 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2022-07-18T03:01:46.833 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:46.833 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2022-07-18T03:01:46.938 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:46.938 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2022-07-18T03:01:47.012 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:47.013 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2022-07-18T03:01:47.081 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:47.082 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2022-07-18T03:01:47.147 INFO:tasks.ceph:Adding keys to all mons... 2022-07-18T03:01:47.148 DEBUG:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/c1.keyring 2022-07-18T03:01:47.244 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2022-07-18T03:01:47.244 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQARzdRi8CvaCRAAgFskyHnsgYmb9eWUJ2OWXQ== 2022-07-18T03:01:47.245 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2022-07-18T03:01:47.245 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQATzdRi1OkdORAAlO5+xInDFY+xtwO91CvnMw== 2022-07-18T03:01:47.249 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2022-07-18T03:01:47.250 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAWzdRi88KbDhAAG6h1FMVDyY0yId+LCxM20w== 2022-07-18T03:01:47.250 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2022-07-18T03:01:47.251 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAYzdRiS7zrHRAACBHla2t3fdMdyzZ49fv2Hg== 2022-07-18T03:01:47.251 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2022-07-18T03:01:47.252 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQARzdRizNnnDBAALhiYJlnlb01CW8I8ZMzIbQ== 2022-07-18T03:01:47.252 INFO:teuthology.orchestra.run.smithi133.stdout:[client.1] 2022-07-18T03:01:47.252 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQARzdRibSjhEhAAB3Sf2rWBMSqRnLwTxDJoaQ== 2022-07-18T03:01:47.253 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-18T03:01:47.344 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-07-18T03:01:47.448 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-07-18T03:01:47.537 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-07-18T03:01:47.591 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-18T03:01:47.695 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-18T03:01:47.818 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-07-18T03:01:47.818 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2022-07-18T03:01:47.932 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2022-07-18T03:01:48.062 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2022-07-18T03:01:48.095 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/c1.monmap 2022-07-18T03:01:48.174 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2022-07-18T03:01:48.175 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2022-07-18T03:01:48.203 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2022-07-18T03:01:48.233 INFO:tasks.ceph.c1.mon.a:Started 2022-07-18T03:01:48.233 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2022-07-18T03:01:48.234 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2022-07-18T03:01:48.244 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2022-07-18T03:01:48.286 INFO:tasks.ceph.c1.mgr.x:Started 2022-07-18T03:01:48.288 INFO:tasks.ceph:Setting crush tunables to default 2022-07-18T03:01:48.289 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c1 osd crush tunables default 2022-07-18T03:01:48.502 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2022-07-18T03:01:48.533 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2022-07-18T03:01:48.534 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:48.535 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2022-07-18T03:01:48.573 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:48.573 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2022-07-18T03:01:48.660 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:01:48.660 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2022-07-18T03:01:48.754 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c1 osd new 977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9 0 2022-07-18T03:01:48.758 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:48.730+0000 7f111e8fe080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-18T03:01:48.826 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:48.825+0000 7f111e8fe080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-18T03:01:48.897 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:48.895+0000 7f111e8fe080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-18T03:01:49.003 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:48.994+0000 7f111e8fe080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-18T03:01:49.095 INFO:teuthology.orchestra.run.smithi133.stdout:0 2022-07-18T03:01:49.153 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c1 osd new 41d2efa6-303e-4511-94b8-c6bb77fcf540 1 2022-07-18T03:01:49.188 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:49.175+0000 7f111e8fe080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-18T03:01:49.249 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:49.246+0000 7f111e8fe080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-18T03:01:49.349 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:49.326+0000 7f111e8fe080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-18T03:01:49.397 INFO:teuthology.orchestra.run.smithi133.stdout:1 2022-07-18T03:01:49.420 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c1 osd new 7b26ded5-dfc6-4052-be1d-4a7ba3c5592c 2 2022-07-18T03:01:49.638 INFO:teuthology.orchestra.run.smithi133.stdout:2 2022-07-18T03:01:49.643 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2022-07-18T03:01:49.643 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2022-07-18T03:01:49.648 INFO:tasks.ceph.c1.osd.0:Started 2022-07-18T03:01:49.649 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2022-07-18T03:01:49.650 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2022-07-18T03:01:49.653 INFO:tasks.ceph.c1.osd.1:Started 2022-07-18T03:01:49.654 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2022-07-18T03:01:49.655 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2022-07-18T03:01:49.663 INFO:tasks.ceph.c1.osd.2:Started 2022-07-18T03:01:49.699 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-07-18T03:01:49.915 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:01:49.915 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":5,"fsid":"c640de7a-c5e8-4a81-b782-87c1207bd4be","created":"2022-07-18T03:01:48.329166+0000","modified":"2022-07-18T03:01:49.628985+0000","last_up_change":"0.000000","last_in_change":"2022-07-18T03:01:49.628985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41d2efa6-303e-4511-94b8-c6bb77fcf540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7b26ded5-dfc6-4052-be1d-4a7ba3c5592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:01:49.935 INFO:tasks.ceph.ceph_manager.c1:[] 2022-07-18T03:01:49.936 INFO:tasks.ceph:Waiting for OSDs to come up 2022-07-18T03:01:49.973 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:49.972+0000 7f111e8fe080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-18T03:01:50.057 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:50.046+0000 7f111e8fe080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-18T03:01:50.273 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-07-18T03:01:50.276 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T03:01:50.254+0000 7f769df1d440 -1 Falling back to public interface 2022-07-18T03:01:50.277 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T03:01:50.258+0000 7fbce83f8440 -1 Falling back to public interface 2022-07-18T03:01:50.311 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T03:01:50.267+0000 7fec17a1c440 -1 Falling back to public interface 2022-07-18T03:01:50.382 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:50.381+0000 7f111e8fe080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-18T03:01:50.524 INFO:teuthology.misc.health.smithi133.stdout: 2022-07-18T03:01:50.525 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":5,"fsid":"c640de7a-c5e8-4a81-b782-87c1207bd4be","created":"2022-07-18T03:01:48.329166+0000","modified":"2022-07-18T03:01:49.628985+0000","last_up_change":"0.000000","last_in_change":"2022-07-18T03:01:49.628985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41d2efa6-303e-4511-94b8-c6bb77fcf540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7b26ded5-dfc6-4052-be1d-4a7ba3c5592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:01:50.544 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-07-18T03:01:50.822 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:50.821+0000 7f111e8fe080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-18T03:01:50.899 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:50.890+0000 7f111e8fe080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-18T03:01:50.962 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:50.956+0000 7f111e8fe080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-18T03:01:51.099 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:51.098+0000 7f111e8fe080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-18T03:01:51.360 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:51.358+0000 7f111e8fe080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-18T03:01:51.495 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:51.488+0000 7f111e8fe080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-18T03:01:51.585 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:51.583+0000 7f111e8fe080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-18T03:01:51.683 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:51.659+0000 7f111e8fe080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-18T03:01:51.760 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:51.755+0000 7f111e8fe080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-18T03:01:52.450 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:52.448+0000 7f111e8fe080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-18T03:01:53.042 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T03:01:53.037+0000 7f769df1d440 -1 osd.0 0 log_to_monitors true 2022-07-18T03:01:53.050 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T03:01:53.044+0000 7fbce83f8440 -1 osd.1 0 log_to_monitors true 2022-07-18T03:01:53.088 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:53.082+0000 7f111e8fe080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-18T03:01:53.308 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:53.307+0000 7f111e8fe080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-18T03:01:53.423 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:53.421+0000 7f111e8fe080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-18T03:01:53.567 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T03:01:53.561+0000 7f111e8fe080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-18T03:01:54.918 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T03:01:54.913+0000 7f7691ccc700 -1 osd.0 0 waiting for initial osdmap 2022-07-18T03:01:54.930 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T03:01:54.924+0000 7f768a44b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-18T03:01:54.992 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T03:01:54.955+0000 7fbcdc1a7700 -1 osd.1 0 waiting for initial osdmap 2022-07-18T03:01:54.992 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T03:01:54.969+0000 7fbcd4926700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-18T03:01:56.878 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-07-18T03:01:57.224 INFO:teuthology.misc.health.smithi133.stdout: 2022-07-18T03:01:57.225 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":9,"fsid":"c640de7a-c5e8-4a81-b782-87c1207bd4be","created":"2022-07-18T03:01:48.329166+0000","modified":"2022-07-18T03:01:56.913454+0000","last_up_change":"2022-07-18T03:01:55.912756+0000","last_in_change":"2022-07-18T03:01:49.628985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":40729},{"type":"v1","addr":"172.21.15.133:6801","nonce":40729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":40729},{"type":"v1","addr":"172.21.15.133:6803","nonce":40729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":40729},{"type":"v1","addr":"172.21.15.133:6807","nonce":40729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":40729},{"type":"v1","addr":"172.21.15.133:6805","nonce":40729}]},"public_addr":"172.21.15.133:6801/40729","cluster_addr":"172.21.15.133:6803/40729","heartbeat_back_addr":"172.21.15.133:6807/40729","heartbeat_front_addr":"172.21.15.133:6805/40729","state":["exists","up"]},{"osd":1,"uuid":"41d2efa6-303e-4511-94b8-c6bb77fcf540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":40734},{"type":"v1","addr":"172.21.15.133:6809","nonce":40734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":40734},{"type":"v1","addr":"172.21.15.133:6811","nonce":40734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":40734},{"type":"v1","addr":"172.21.15.133:6815","nonce":40734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":40734},{"type":"v1","addr":"172.21.15.133:6813","nonce":40734}]},"public_addr":"172.21.15.133:6809/40734","cluster_addr":"172.21.15.133:6811/40734","heartbeat_back_addr":"172.21.15.133:6815/40734","heartbeat_front_addr":"172.21.15.133:6813/40734","state":["exists","up"]},{"osd":2,"uuid":"7b26ded5-dfc6-4052-be1d-4a7ba3c5592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:53.998123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:54.010078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:01:57.241 DEBUG:teuthology.misc:2 of 3 OSDs are up 2022-07-18T03:01:58.018 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T03:01:58.016+0000 7fec17a1c440 -1 osd.2 0 log_to_monitors true 2022-07-18T03:01:59.581 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T03:01:59.578+0000 7fec0b7cb700 -1 osd.2 0 waiting for initial osdmap 2022-07-18T03:01:59.621 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T03:01:59.594+0000 7fec03f4a700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-18T03:02:03.579 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-07-18T03:02:03.907 INFO:teuthology.misc.health.smithi133.stdout: 2022-07-18T03:02:03.908 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":13,"fsid":"c640de7a-c5e8-4a81-b782-87c1207bd4be","created":"2022-07-18T03:01:48.329166+0000","modified":"2022-07-18T03:02:01.580854+0000","last_up_change":"2022-07-18T03:02:00.577556+0000","last_in_change":"2022-07-18T03:01:49.628985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-18T03:01:57.620198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":40729},{"type":"v1","addr":"172.21.15.133:6801","nonce":40729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":40729},{"type":"v1","addr":"172.21.15.133:6803","nonce":40729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":40729},{"type":"v1","addr":"172.21.15.133:6807","nonce":40729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":40729},{"type":"v1","addr":"172.21.15.133:6805","nonce":40729}]},"public_addr":"172.21.15.133:6801/40729","cluster_addr":"172.21.15.133:6803/40729","heartbeat_back_addr":"172.21.15.133:6807/40729","heartbeat_front_addr":"172.21.15.133:6805/40729","state":["exists","up"]},{"osd":1,"uuid":"41d2efa6-303e-4511-94b8-c6bb77fcf540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":40734},{"type":"v1","addr":"172.21.15.133:6809","nonce":40734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":40734},{"type":"v1","addr":"172.21.15.133:6811","nonce":40734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":40734},{"type":"v1","addr":"172.21.15.133:6815","nonce":40734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":40734},{"type":"v1","addr":"172.21.15.133:6813","nonce":40734}]},"public_addr":"172.21.15.133:6809/40734","cluster_addr":"172.21.15.133:6811/40734","heartbeat_back_addr":"172.21.15.133:6815/40734","heartbeat_front_addr":"172.21.15.133:6813/40734","state":["exists","up"]},{"osd":2,"uuid":"7b26ded5-dfc6-4052-be1d-4a7ba3c5592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":40746},{"type":"v1","addr":"172.21.15.133:6817","nonce":40746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":40746},{"type":"v1","addr":"172.21.15.133:6819","nonce":40746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":40746},{"type":"v1","addr":"172.21.15.133:6823","nonce":40746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":40746},{"type":"v1","addr":"172.21.15.133:6821","nonce":40746}]},"public_addr":"172.21.15.133:6817/40746","cluster_addr":"172.21.15.133:6819/40746","heartbeat_back_addr":"172.21.15.133:6823/40746","heartbeat_front_addr":"172.21.15.133:6821/40746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:53.998123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:54.010078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:59.050393+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:03.932 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-07-18T03:02:03.932 INFO:tasks.ceph:Creating RBD pool 2022-07-18T03:02:03.933 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c1 osd pool create rbd 8 2022-07-18T03:02:04.608 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'rbd' created 2022-07-18T03:02:04.608 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-07-18T03:02:05.601 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rbd' on pool 'rbd' 2022-07-18T03:02:05.617 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2022-07-18T03:02:05.618 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2022-07-18T03:02:05.619 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2022-07-18T03:02:05.620 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2022-07-18T03:02:05.621 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-07-18T03:02:06.001 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:06.034 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":40377},{"type":"v1","addr":"172.21.15.133:6825","nonce":40377}]},"active_addr":"172.21.15.133:6825/40377","active_change":"2022-07-18T03:01:53.576028+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":3438980968}]},{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":1670977431}]},{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":1954857049}]},{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":3004014966}]}]}} 2022-07-18T03:02:06.043 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2022-07-18T03:02:06.043 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2022-07-18T03:02:06.044 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-07-18T03:02:06.388 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:06.388 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":15,"fsid":"c640de7a-c5e8-4a81-b782-87c1207bd4be","created":"2022-07-18T03:01:48.329166+0000","modified":"2022-07-18T03:02:05.588910+0000","last_up_change":"2022-07-18T03:02:00.577556+0000","last_in_change":"2022-07-18T03:01:49.628985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-18T03:01:57.620198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-18T03:02:04.262175+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":40729},{"type":"v1","addr":"172.21.15.133:6801","nonce":40729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":40729},{"type":"v1","addr":"172.21.15.133:6803","nonce":40729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":40729},{"type":"v1","addr":"172.21.15.133:6807","nonce":40729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":40729},{"type":"v1","addr":"172.21.15.133:6805","nonce":40729}]},"public_addr":"172.21.15.133:6801/40729","cluster_addr":"172.21.15.133:6803/40729","heartbeat_back_addr":"172.21.15.133:6807/40729","heartbeat_front_addr":"172.21.15.133:6805/40729","state":["exists","up"]},{"osd":1,"uuid":"41d2efa6-303e-4511-94b8-c6bb77fcf540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":40734},{"type":"v1","addr":"172.21.15.133:6809","nonce":40734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":40734},{"type":"v1","addr":"172.21.15.133:6811","nonce":40734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":40734},{"type":"v1","addr":"172.21.15.133:6815","nonce":40734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":40734},{"type":"v1","addr":"172.21.15.133:6813","nonce":40734}]},"public_addr":"172.21.15.133:6809/40734","cluster_addr":"172.21.15.133:6811/40734","heartbeat_back_addr":"172.21.15.133:6815/40734","heartbeat_front_addr":"172.21.15.133:6813/40734","state":["exists","up"]},{"osd":2,"uuid":"7b26ded5-dfc6-4052-be1d-4a7ba3c5592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":40746},{"type":"v1","addr":"172.21.15.133:6817","nonce":40746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":40746},{"type":"v1","addr":"172.21.15.133:6819","nonce":40746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":40746},{"type":"v1","addr":"172.21.15.133:6823","nonce":40746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":40746},{"type":"v1","addr":"172.21.15.133:6821","nonce":40746}]},"public_addr":"172.21.15.133:6817/40746","cluster_addr":"172.21.15.133:6819/40746","heartbeat_back_addr":"172.21.15.133:6823/40746","heartbeat_front_addr":"172.21.15.133:6821/40746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:53.998123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:54.010078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:59.050393+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:06.389 INFO:tasks.ceph.ceph_manager.c1:all up! 2022-07-18T03:02:06.406 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-07-18T03:02:06.722 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:06.723 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":16,"fsid":"c640de7a-c5e8-4a81-b782-87c1207bd4be","created":"2022-07-18T03:01:48.329166+0000","modified":"2022-07-18T03:02:06.591992+0000","last_up_change":"2022-07-18T03:02:00.577556+0000","last_in_change":"2022-07-18T03:01:49.628985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-18T03:01:57.620198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-18T03:02:04.262175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"977c20d9-cd13-4bfa-a9b2-c6cfd3b81af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":40729},{"type":"v1","addr":"172.21.15.133:6801","nonce":40729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":40729},{"type":"v1","addr":"172.21.15.133:6803","nonce":40729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":40729},{"type":"v1","addr":"172.21.15.133:6807","nonce":40729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":40729},{"type":"v1","addr":"172.21.15.133:6805","nonce":40729}]},"public_addr":"172.21.15.133:6801/40729","cluster_addr":"172.21.15.133:6803/40729","heartbeat_back_addr":"172.21.15.133:6807/40729","heartbeat_front_addr":"172.21.15.133:6805/40729","state":["exists","up"]},{"osd":1,"uuid":"41d2efa6-303e-4511-94b8-c6bb77fcf540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":40734},{"type":"v1","addr":"172.21.15.133:6809","nonce":40734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":40734},{"type":"v1","addr":"172.21.15.133:6811","nonce":40734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":40734},{"type":"v1","addr":"172.21.15.133:6815","nonce":40734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":40734},{"type":"v1","addr":"172.21.15.133:6813","nonce":40734}]},"public_addr":"172.21.15.133:6809/40734","cluster_addr":"172.21.15.133:6811/40734","heartbeat_back_addr":"172.21.15.133:6815/40734","heartbeat_front_addr":"172.21.15.133:6813/40734","state":["exists","up"]},{"osd":2,"uuid":"7b26ded5-dfc6-4052-be1d-4a7ba3c5592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":40746},{"type":"v1","addr":"172.21.15.133:6817","nonce":40746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":40746},{"type":"v1","addr":"172.21.15.133:6819","nonce":40746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":40746},{"type":"v1","addr":"172.21.15.133:6823","nonce":40746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":40746},{"type":"v1","addr":"172.21.15.133:6821","nonce":40746}]},"public_addr":"172.21.15.133:6817/40746","cluster_addr":"172.21.15.133:6819/40746","heartbeat_back_addr":"172.21.15.133:6823/40746","heartbeat_front_addr":"172.21.15.133:6821/40746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:53.998123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:54.010078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:01:59.050393+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:06.737 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2022-07-18T03:02:06.741 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2022-07-18T03:02:06.742 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2022-07-18T03:02:06.969 INFO:teuthology.orchestra.run.smithi133.stdout:34359738374 2022-07-18T03:02:06.974 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-07-18T03:02:06.976 INFO:teuthology.orchestra.run.smithi133.stdout:51539607557 2022-07-18T03:02:06.977 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-07-18T03:02:07.098 INFO:teuthology.orchestra.run.smithi133.stdout:34359738374 2022-07-18T03:02:07.099 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-07-18T03:02:07.479 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2022-07-18T03:02:07.480 INFO:teuthology.orchestra.run.smithi133.stdout:51539607555 2022-07-18T03:02:07.496 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738372 for osd.1 2022-07-18T03:02:07.497 INFO:tasks.ceph.ceph_manager.c1:need seq 51539607557 got 51539607555 for osd.2 2022-07-18T03:02:07.536 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2022-07-18T03:02:07.550 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738372 for osd.0 2022-07-18T03:02:08.508 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-07-18T03:02:08.509 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-07-18T03:02:08.556 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-07-18T03:02:08.866 INFO:teuthology.orchestra.run.smithi133.stdout:51539607557 2022-07-18T03:02:08.878 INFO:teuthology.orchestra.run.smithi133.stdout:34359738374 2022-07-18T03:02:08.880 INFO:tasks.ceph.ceph_manager.c1:need seq 51539607557 got 51539607557 for osd.2 2022-07-18T03:02:08.880 DEBUG:teuthology.parallel:result is None 2022-07-18T03:02:08.893 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738374 for osd.1 2022-07-18T03:02:08.893 DEBUG:teuthology.parallel:result is None 2022-07-18T03:02:08.938 INFO:teuthology.orchestra.run.smithi133.stdout:34359738374 2022-07-18T03:02:08.963 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738374 for osd.0 2022-07-18T03:02:08.963 DEBUG:teuthology.parallel:result is None 2022-07-18T03:02:08.980 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2022-07-18T03:02:08.981 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-07-18T03:02:09.301 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:09.302 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2022-07-18T03:02:09.308 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2022-07-18T03:02:07.582351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25000,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547800,"statfs":{"total":322122547200,"available":322096947200,"internally_reserved":0,"allocated":1589248,"data_stored":1292991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000885"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.597527+0000","last_change":"2022-07-18T03:02:05.597527+0000","last_active":"2022-07-18T03:02:05.597527+0000","last_peered":"2022-07-18T03:02:05.597527+0000","last_clean":"2022-07-18T03:02:05.597527+0000","last_became_active":"2022-07-18T03:02:05.597255+0000","last_became_peered":"2022-07-18T03:02:05.597255+0000","last_unstale":"2022-07-18T03:02:05.597527+0000","last_undegraded":"2022-07-18T03:02:05.597527+0000","last_fullsized":"2022-07-18T03:02:05.597527+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T14:55:13.411871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.598681+0000","last_change":"2022-07-18T03:02:05.598681+0000","last_active":"2022-07-18T03:02:05.598681+0000","last_peered":"2022-07-18T03:02:05.598681+0000","last_clean":"2022-07-18T03:02:05.598681+0000","last_became_active":"2022-07-18T03:02:05.598408+0000","last_became_peered":"2022-07-18T03:02:05.598408+0000","last_unstale":"2022-07-18T03:02:05.598681+0000","last_undegraded":"2022-07-18T03:02:05.598681+0000","last_fullsized":"2022-07-18T03:02:05.598681+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T09:18:25.442453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.596937+0000","last_change":"2022-07-18T03:02:05.596937+0000","last_active":"2022-07-18T03:02:05.596937+0000","last_peered":"2022-07-18T03:02:05.596937+0000","last_clean":"2022-07-18T03:02:05.596937+0000","last_became_active":"2022-07-18T03:02:05.596664+0000","last_became_peered":"2022-07-18T03:02:05.596664+0000","last_unstale":"2022-07-18T03:02:05.596937+0000","last_undegraded":"2022-07-18T03:02:05.596937+0000","last_fullsized":"2022-07-18T03:02:05.596937+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T09:03:53.812459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.597739+0000","last_change":"2022-07-18T03:02:05.597739+0000","last_active":"2022-07-18T03:02:05.597739+0000","last_peered":"2022-07-18T03:02:05.597739+0000","last_clean":"2022-07-18T03:02:05.597739+0000","last_became_active":"2022-07-18T03:02:05.597499+0000","last_became_peered":"2022-07-18T03:02:05.597499+0000","last_unstale":"2022-07-18T03:02:05.597739+0000","last_undegraded":"2022-07-18T03:02:05.597739+0000","last_fullsized":"2022-07-18T03:02:05.597739+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T03:25:16.373728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-07-18T03:02:04.590286+0000","last_change":"2022-07-18T03:02:00.028797+0000","last_active":"2022-07-18T03:02:04.590286+0000","last_peered":"2022-07-18T03:02:04.590286+0000","last_clean":"2022-07-18T03:02:04.590286+0000","last_became_active":"2022-07-18T03:02:00.028092+0000","last_became_peered":"2022-07-18T03:02:00.028092+0000","last_unstale":"2022-07-18T03:02:04.590286+0000","last_undegraded":"2022-07-18T03:02:04.590286+0000","last_fullsized":"2022-07-18T03:02:04.590286+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:01:58.102798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:01:58.102798+0000","last_clean_scrub_stamp":"2022-07-18T03:01:58.102798+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T11:28:57.067732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.601833+0000","last_change":"2022-07-18T03:02:05.601833+0000","last_active":"2022-07-18T03:02:05.601833+0000","last_peered":"2022-07-18T03:02:05.601833+0000","last_clean":"2022-07-18T03:02:05.601833+0000","last_became_active":"2022-07-18T03:02:05.601598+0000","last_became_peered":"2022-07-18T03:02:05.601598+0000","last_unstale":"2022-07-18T03:02:05.601833+0000","last_undegraded":"2022-07-18T03:02:05.601833+0000","last_fullsized":"2022-07-18T03:02:05.601833+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T09:17:58.961365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.598231+0000","last_change":"2022-07-18T03:02:05.598231+0000","last_active":"2022-07-18T03:02:05.598231+0000","last_peered":"2022-07-18T03:02:05.598231+0000","last_clean":"2022-07-18T03:02:05.598231+0000","last_became_active":"2022-07-18T03:02:05.597985+0000","last_became_peered":"2022-07-18T03:02:05.597985+0000","last_unstale":"2022-07-18T03:02:05.598231+0000","last_undegraded":"2022-07-18T03:02:05.598231+0000","last_fullsized":"2022-07-18T03:02:05.598231+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T14:17:20.707219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.603958+0000","last_change":"2022-07-18T03:02:05.599086+0000","last_active":"2022-07-18T03:02:05.603958+0000","last_peered":"2022-07-18T03:02:05.603958+0000","last_clean":"2022-07-18T03:02:05.603958+0000","last_became_active":"2022-07-18T03:02:05.598829+0000","last_became_peered":"2022-07-18T03:02:05.598829+0000","last_unstale":"2022-07-18T03:02:05.603958+0000","last_undegraded":"2022-07-18T03:02:05.603958+0000","last_fullsized":"2022-07-18T03:02:05.603958+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T12:59:48.789259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.599228+0000","last_change":"2022-07-18T03:02:05.599228+0000","last_active":"2022-07-18T03:02:05.599228+0000","last_peered":"2022-07-18T03:02:05.599228+0000","last_clean":"2022-07-18T03:02:05.599228+0000","last_became_active":"2022-07-18T03:02:05.598968+0000","last_became_peered":"2022-07-18T03:02:05.598968+0000","last_unstale":"2022-07-18T03:02:05.599228+0000","last_undegraded":"2022-07-18T03:02:05.599228+0000","last_fullsized":"2022-07-18T03:02:05.599228+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T07:44:31.087736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7688,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849912,"statfs":{"total":107374182400,"available":107366309888,"internally_reserved":0,"allocated":131072,"data_stored":36263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8660,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848940,"statfs":{"total":107374182400,"available":107365314560,"internally_reserved":0,"allocated":733184,"data_stored":630097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8652,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848948,"statfs":{"total":107374182400,"available":107365322752,"internally_reserved":0,"allocated":724992,"data_stored":626631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-18T03:02:09.312 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-07-18T03:02:09.621 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:09.622 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2022-07-18T03:02:09.700 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2022-07-18T03:02:09.582703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25000,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547800,"statfs":{"total":322122547200,"available":322096947200,"internally_reserved":0,"allocated":1589248,"data_stored":1292991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001236"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.597527+0000","last_change":"2022-07-18T03:02:05.597527+0000","last_active":"2022-07-18T03:02:05.597527+0000","last_peered":"2022-07-18T03:02:05.597527+0000","last_clean":"2022-07-18T03:02:05.597527+0000","last_became_active":"2022-07-18T03:02:05.597255+0000","last_became_peered":"2022-07-18T03:02:05.597255+0000","last_unstale":"2022-07-18T03:02:05.597527+0000","last_undegraded":"2022-07-18T03:02:05.597527+0000","last_fullsized":"2022-07-18T03:02:05.597527+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T14:55:13.411871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.598681+0000","last_change":"2022-07-18T03:02:05.598681+0000","last_active":"2022-07-18T03:02:05.598681+0000","last_peered":"2022-07-18T03:02:05.598681+0000","last_clean":"2022-07-18T03:02:05.598681+0000","last_became_active":"2022-07-18T03:02:05.598408+0000","last_became_peered":"2022-07-18T03:02:05.598408+0000","last_unstale":"2022-07-18T03:02:05.598681+0000","last_undegraded":"2022-07-18T03:02:05.598681+0000","last_fullsized":"2022-07-18T03:02:05.598681+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T09:18:25.442453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.596937+0000","last_change":"2022-07-18T03:02:05.596937+0000","last_active":"2022-07-18T03:02:05.596937+0000","last_peered":"2022-07-18T03:02:05.596937+0000","last_clean":"2022-07-18T03:02:05.596937+0000","last_became_active":"2022-07-18T03:02:05.596664+0000","last_became_peered":"2022-07-18T03:02:05.596664+0000","last_unstale":"2022-07-18T03:02:05.596937+0000","last_undegraded":"2022-07-18T03:02:05.596937+0000","last_fullsized":"2022-07-18T03:02:05.596937+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T09:03:53.812459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.597739+0000","last_change":"2022-07-18T03:02:05.597739+0000","last_active":"2022-07-18T03:02:05.597739+0000","last_peered":"2022-07-18T03:02:05.597739+0000","last_clean":"2022-07-18T03:02:05.597739+0000","last_became_active":"2022-07-18T03:02:05.597499+0000","last_became_peered":"2022-07-18T03:02:05.597499+0000","last_unstale":"2022-07-18T03:02:05.597739+0000","last_undegraded":"2022-07-18T03:02:05.597739+0000","last_fullsized":"2022-07-18T03:02:05.597739+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T03:25:16.373728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-07-18T03:02:04.590286+0000","last_change":"2022-07-18T03:02:00.028797+0000","last_active":"2022-07-18T03:02:04.590286+0000","last_peered":"2022-07-18T03:02:04.590286+0000","last_clean":"2022-07-18T03:02:04.590286+0000","last_became_active":"2022-07-18T03:02:00.028092+0000","last_became_peered":"2022-07-18T03:02:00.028092+0000","last_unstale":"2022-07-18T03:02:04.590286+0000","last_undegraded":"2022-07-18T03:02:04.590286+0000","last_fullsized":"2022-07-18T03:02:04.590286+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:01:58.102798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:01:58.102798+0000","last_clean_scrub_stamp":"2022-07-18T03:01:58.102798+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T11:28:57.067732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.601833+0000","last_change":"2022-07-18T03:02:05.601833+0000","last_active":"2022-07-18T03:02:05.601833+0000","last_peered":"2022-07-18T03:02:05.601833+0000","last_clean":"2022-07-18T03:02:05.601833+0000","last_became_active":"2022-07-18T03:02:05.601598+0000","last_became_peered":"2022-07-18T03:02:05.601598+0000","last_unstale":"2022-07-18T03:02:05.601833+0000","last_undegraded":"2022-07-18T03:02:05.601833+0000","last_fullsized":"2022-07-18T03:02:05.601833+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T09:17:58.961365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.598231+0000","last_change":"2022-07-18T03:02:05.598231+0000","last_active":"2022-07-18T03:02:05.598231+0000","last_peered":"2022-07-18T03:02:05.598231+0000","last_clean":"2022-07-18T03:02:05.598231+0000","last_became_active":"2022-07-18T03:02:05.597985+0000","last_became_peered":"2022-07-18T03:02:05.597985+0000","last_unstale":"2022-07-18T03:02:05.598231+0000","last_undegraded":"2022-07-18T03:02:05.598231+0000","last_fullsized":"2022-07-18T03:02:05.598231+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T14:17:20.707219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.603958+0000","last_change":"2022-07-18T03:02:05.599086+0000","last_active":"2022-07-18T03:02:05.603958+0000","last_peered":"2022-07-18T03:02:05.603958+0000","last_clean":"2022-07-18T03:02:05.603958+0000","last_became_active":"2022-07-18T03:02:05.598829+0000","last_became_peered":"2022-07-18T03:02:05.598829+0000","last_unstale":"2022-07-18T03:02:05.603958+0000","last_undegraded":"2022-07-18T03:02:05.603958+0000","last_fullsized":"2022-07-18T03:02:05.603958+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T12:59:48.789259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":15,"state":"active+clean","last_fresh":"2022-07-18T03:02:05.599228+0000","last_change":"2022-07-18T03:02:05.599228+0000","last_active":"2022-07-18T03:02:05.599228+0000","last_peered":"2022-07-18T03:02:05.599228+0000","last_clean":"2022-07-18T03:02:05.599228+0000","last_became_active":"2022-07-18T03:02:05.598968+0000","last_became_peered":"2022-07-18T03:02:05.598968+0000","last_unstale":"2022-07-18T03:02:05.599228+0000","last_undegraded":"2022-07-18T03:02:05.599228+0000","last_fullsized":"2022-07-18T03:02:05.599228+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:04.585380+0000","last_clean_scrub_stamp":"2022-07-18T03:02:04.585380+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T07:44:31.087736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7688,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849912,"statfs":{"total":107374182400,"available":107366309888,"internally_reserved":0,"allocated":131072,"data_stored":36263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8660,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848940,"statfs":{"total":107374182400,"available":107365314560,"internally_reserved":0,"allocated":733184,"data_stored":630097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8652,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848948,"statfs":{"total":107374182400,"available":107365322752,"internally_reserved":0,"allocated":724992,"data_stored":626631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-18T03:02:09.702 INFO:tasks.ceph.ceph_manager.c1:clean! 2022-07-18T03:02:09.704 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2022-07-18T03:02:09.707 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2022-07-18T03:02:09.708 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2022-07-18T03:02:10.078 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:10.079 INFO:teuthology.orchestra.run.smithi133.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-18T03:02:10.120 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2022-07-18T03:02:10.121 INFO:teuthology.run_tasks:Running task ceph... 2022-07-18T03:02:10.187 INFO:tasks.ceph:Creating ceph cluster c2... 2022-07-18T03:02:10.187 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-07-18T03:02:10.195 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi/6935300', 'branch': 'wip-rgwlc-azone', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6935300', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mbenjamin-2022-07-17_22:24:15-rgw-wip-rgwlc-azone-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-rgwlc-azone'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-rgwlc-azone', 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017'}}, 'owner': 'scheduled_mbenjamin@shatner', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgwlc-azone', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'targets': {'smithi133.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbz643aJzTA1MGouCBCaX2wAlr7gxhsuzbiWpPfB1J+XwHbAt+kDOGp2Ft/jOzaXFKrq9e20ENTcd5lzC730MsuhxkXAFpXApXKyIfdGdowsKOGictcs22h7dXfvGGl7lytZg65AHT5kAGQZkADb24rTTvlM9ZtQrM2wLMo5pHBM4p+2nlHFEMH4NvWM0W4qQ+MUTv+V9EbLfB3Kf4OJcfMuw4S9+fSDf3bqVb8I7WIx/wtYFxEEJT4PolE/dvR7kCe/C6rqqqDMP154xs5/17c652VftNZq41Vv8IueKmd3wq2ZOM2zqrNLCGr1FkE7T4jBC89zWOCn9PRu+RF9Vl0lPLKOrkVOVOoQdEZSZixPiOEkRyKEkWNxkyIPtS/EDTjjDKIZSrCeRLm3EOXMP0eFTEa98qWrsoaxtLOeDIPM3HwufNX5YU5DLyvmx+x9ZQilp3QwJD89g+dzQqSSdDgxjTvJ2ZDvid4MkWd8G5Faw/UCUtHNq8FLc9BCGZ+pM=', 'smithi159.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0XX/4wSzWw+RBIBAqfnofJBXzIjtevgLHZtgd1wszCkZ2NrfTAt7OZWuj/Imft/uz6GkoqE0v5k/VRYEThvU4Fg0FCIKaBPOc7rZOR3Ego4Qk+Xun+Fm0AfDMCVtZTsCY/dImi9Kh2bOruLyGZnRPD4hnLoYNgsHioSMJjGOyWj5cIzGU7u/UNmmfSDvQqTnBUbvXYWLuRkwYDnyUGt36BVOM+OHhS3ODwnKXtNycReRYuxdoh8T4Q193sdYXaaISZxtQnPiIG/tRMy7vDanx5m+m9I0i65kh/J4umQQuqQ/DutpXO8TrPXAxqYtkyloh9IrBRV1bwN55ziIA9cPLHl9LBFA7cGKjk/2Bm5tEz9DxeMWUPygu6jFc+UUh3zNTaaJp9nSPMYmwYP8Ub7TmEluVDqPJh6jctHSYllsA/scLg19rIGTi7E1s7KxLOu8bBT++U9T6HPeVXqwZqZlLmEoIBKTE2wnIS5NHj/M/pdzfAvd/OqdUsOJGxMTJHDc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'ff10064ab878e58afb1d2538b2086a2d4a41c017', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'args': ['-a', '!bucket_sync_disable,!bucket_reshard'], 'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'main', 'teuthology_sha1': '89d874d555f56b1c1f383df72487aecec0ee6cf9', 'timestamp': '2022-07-17_22:24:15', 'tube': 'smithi', 'user': 'mbenjamin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-07-18T03:02:10.206 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-07-18T03:02:10.230 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-07-18T03:02:10.323 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2022-07-18T03:02:10.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2022-07-18T03:02:10.421 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:10.421 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2022-07-18T03:02:10.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-18T03:02:10.537 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-18T03:02:10.593 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-18T03:02:10.594 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:02:10.615 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 122341 Links: 1 2022-07-18T03:02:10.615 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:02:10.616 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:02:10.616 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2022-07-18 03:01:15.223637505 +0000 2022-07-18T03:02:10.617 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2022-07-18 02:57:54.886412524 +0000 2022-07-18T03:02:10.618 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2022-07-18 02:57:54.886412524 +0000 2022-07-18T03:02:10.618 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2022-07-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2022-07-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2022-07-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000240391 s, 2.1 MB/s 2022-07-18T03:02:10.748 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-18T03:02:10.829 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-18T03:02:10.829 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:02:10.829 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 126106 Links: 1 2022-07-18T03:02:10.830 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:02:10.864 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:02:10.865 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2022-07-18 03:01:15.223637505 +0000 2022-07-18T03:02:10.870 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2022-07-18 02:57:55.588410429 +0000 2022-07-18T03:02:10.871 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2022-07-18 02:57:55.588410429 +0000 2022-07-18T03:02:10.871 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2022-07-18T03:02:10.873 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2022-07-18T03:02:10.873 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2022-07-18T03:02:10.878 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000234317 s, 2.2 MB/s 2022-07-18T03:02:10.879 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-18T03:02:10.942 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-18T03:02:10.943 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:02:10.944 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 113338 Links: 1 2022-07-18T03:02:10.944 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:02:10.944 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:02:10.945 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2022-07-18 03:01:15.223637505 +0000 2022-07-18T03:02:10.945 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2022-07-18 02:57:56.261408420 +0000 2022-07-18T03:02:10.955 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2022-07-18 02:57:56.261408420 +0000 2022-07-18T03:02:10.956 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2022-07-18T03:02:10.966 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2022-07-18T03:02:10.966 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2022-07-18T03:02:10.967 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.00024177 s, 2.1 MB/s 2022-07-18T03:02:10.967 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-18T03:02:11.055 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-18T03:02:11.055 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-18T03:02:11.055 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 123613 Links: 1 2022-07-18T03:02:11.056 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-18T03:02:11.056 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2022-07-18T03:02:11.056 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2022-07-18 03:01:15.223637505 +0000 2022-07-18T03:02:11.056 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2022-07-18 02:57:56.891406539 +0000 2022-07-18T03:02:11.056 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2022-07-18 02:57:56.891406539 +0000 2022-07-18T03:02:11.057 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2022-07-18T03:02:11.063 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2022-07-18T03:02:11.063 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2022-07-18T03:02:11.064 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000212928 s, 2.4 MB/s 2022-07-18T03:02:11.064 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:02:11.065 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-07-18T03:02:11.065 INFO:tasks.ceph:Generating config... 2022-07-18T03:02:11.067 INFO:tasks.ceph:[client] debug rgw = 20 2022-07-18T03:02:11.067 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-07-18T03:02:11.068 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-07-18T03:02:11.068 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-07-18T03:02:11.068 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-07-18T03:02:11.068 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-07-18T03:02:11.069 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-07-18T03:02:11.069 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-07-18T03:02:11.070 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-07-18T03:02:11.071 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-07-18T03:02:11.071 INFO:tasks.ceph:[client] setgroup = ceph 2022-07-18T03:02:11.071 INFO:tasks.ceph:[client] setuser = ceph 2022-07-18T03:02:11.072 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-07-18T03:02:11.072 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-07-18T03:02:11.072 INFO:tasks.ceph:[mgr] debug ms = 1 2022-07-18T03:02:11.073 INFO:tasks.ceph:[mon] debug mon = 20 2022-07-18T03:02:11.073 INFO:tasks.ceph:[mon] debug ms = 1 2022-07-18T03:02:11.073 INFO:tasks.ceph:[mon] debug paxos = 20 2022-07-18T03:02:11.073 INFO:tasks.ceph:[osd] debug ms = 1 2022-07-18T03:02:11.114 INFO:tasks.ceph:[osd] debug osd = 20 2022-07-18T03:02:11.118 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-07-18T03:02:11.124 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-07-18T03:02:11.124 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-07-18T03:02:11.125 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-07-18T03:02:11.125 INFO:tasks.ceph:Setting up c2.mon.a... 2022-07-18T03:02:11.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2022-07-18T03:02:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/c2.keyring 2022-07-18T03:02:11.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2022-07-18T03:02:11.313 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-07-18T03:02:11.347 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.159')] 2022-07-18T03:02:11.348 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.159'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-07-18T03:02:11.355 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:11.356 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-07-18T03:02:11.488 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.159 --print /home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:11.574 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:02:11.575 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:02:11.578 INFO:teuthology.orchestra.run.smithi159.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:11.579 INFO:teuthology.orchestra.run.smithi159.stdout:monmaptool: generated fsid c5d94c41-00fc-445f-ba02-75a44483a17a 2022-07-18T03:02:11.580 INFO:teuthology.orchestra.run.smithi159.stdout:setting min_mon_release = octopus 2022-07-18T03:02:11.597 INFO:teuthology.orchestra.run.smithi159.stdout:epoch 0 2022-07-18T03:02:11.598 INFO:teuthology.orchestra.run.smithi159.stdout:fsid c5d94c41-00fc-445f-ba02-75a44483a17a 2022-07-18T03:02:11.598 INFO:teuthology.orchestra.run.smithi159.stdout:last_changed 2022-07-18T03:02:11.573184+0000 2022-07-18T03:02:11.599 INFO:teuthology.orchestra.run.smithi159.stdout:created 2022-07-18T03:02:11.573184+0000 2022-07-18T03:02:11.599 INFO:teuthology.orchestra.run.smithi159.stdout:min_mon_release 15 (octopus) 2022-07-18T03:02:11.600 INFO:teuthology.orchestra.run.smithi159.stdout:election_strategy: 1 2022-07-18T03:02:11.600 INFO:teuthology.orchestra.run.smithi159.stdout:0: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.a 2022-07-18T03:02:11.600 INFO:teuthology.orchestra.run.smithi159.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2022-07-18T03:02:11.601 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-07-18T03:02:11.661 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID c5d94c41-00fc-445f-ba02-75a44483a17a... 2022-07-18T03:02:11.663 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-07-18T03:02:11.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-07-18T03:02:11.721 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2022-07-18T03:02:11.721 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2022-07-18T03:02:11.722 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-18T03:02:11.722 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2022-07-18T03:02:11.723 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.723 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2022-07-18T03:02:11.724 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.724 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2022-07-18T03:02:11.725 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.725 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2022-07-18T03:02:11.725 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2022-07-18T03:02:11.726 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.726 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2022-07-18T03:02:11.727 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2022-07-18T03:02:11.727 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.728 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-18T03:02:11.728 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2022-07-18T03:02:11.728 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.729 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2022-07-18T03:02:11.729 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-18T03:02:11.730 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.730 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2022-07-18T03:02:11.730 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.731 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2022-07-18T03:02:11.731 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2022-07-18T03:02:11.732 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2022-07-18T03:02:11.732 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2022-07-18T03:02:11.733 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2022-07-18T03:02:11.733 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2022-07-18T03:02:11.742 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2022-07-18T03:02:11.743 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-18T03:02:11.743 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-18T03:02:11.775 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2022-07-18T03:02:11.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.776 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-07-18T03:02:11.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.777 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2022-07-18T03:02:11.801 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.813 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2022-07-18T03:02:11.814 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.814 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2022-07-18T03:02:11.815 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2022-07-18T03:02:11.815 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2022-07-18T03:02:11.816 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.159 2022-07-18T03:02:11.821 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = c5d94c41-00fc-445f-ba02-75a44483a17a 2022-07-18T03:02:11.854 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.855 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2022-07-18T03:02:11.855 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2022-07-18T03:02:11.856 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.856 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2022-07-18T03:02:11.857 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2022-07-18T03:02:11.857 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.857 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2022-07-18T03:02:11.898 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2022-07-18T03:02:11.899 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.899 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2022-07-18T03:02:11.900 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2022-07-18T03:02:11.900 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2022-07-18T03:02:11.900 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.901 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2022-07-18T03:02:11.901 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2022-07-18T03:02:11.938 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.939 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2022-07-18T03:02:11.939 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.940 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2022-07-18T03:02:11.940 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.941 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2022-07-18T03:02:11.941 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:11.941 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2022-07-18T03:02:12.011 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2022-07-18T03:02:12.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.012 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2022-07-18T03:02:12.013 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2022-07-18T03:02:12.013 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2022-07-18T03:02:12.014 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 20 2022-07-18T03:02:12.034 INFO:teuthology.orchestra.run.smithi133.stdout: rocksdb delete range threshold = 0 2022-07-18T03:02:12.040 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.040 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2022-07-18T03:02:12.041 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2022-07-18T03:02:12.041 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2022-07-18T03:02:12.052 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2022-07-18T03:02:12.053 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2022-07-18T03:02:12.092 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:02:12.095 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:02:12.095 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2022-07-18T03:02:12.098 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.098 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2022-07-18T03:02:12.098 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2022-07-18T03:02:12.104 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2022-07-18T03:02:12.104 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2022-07-18T03:02:12.104 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2022-07-18T03:02:12.105 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2022-07-18T03:02:12.105 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2022-07-18T03:02:12.105 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:02:12.105 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2022-07-18T03:02:12.105 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2022-07-18T03:02:12.130 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:02:12.130 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.135 INFO:teuthology.orchestra.run.smithi133.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-18T03:02:12.138 INFO:teuthology.orchestra.run.smithi133.stdout: auth mon ticket ttl = 660 # 11m 2022-07-18T03:02:12.138 INFO:teuthology.orchestra.run.smithi133.stdout: auth service ticket ttl = 240 # 4m 2022-07-18T03:02:12.152 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.152 INFO:teuthology.orchestra.run.smithi133.stdout: # don't complain about insecure global_id in the test suite 2022-07-18T03:02:12.153 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-18T03:02:12.153 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-18T03:02:12.153 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.153 INFO:teuthology.orchestra.run.smithi133.stdout: # 1m isn't quite enough 2022-07-18T03:02:12.154 INFO:teuthology.orchestra.run.smithi133.stdout: mon_down_mkfs_grace = 2m 2022-07-18T03:02:12.154 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.154 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_filestore_osds = false 2022-07-18T03:02:12.155 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:02:12.155 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2022-07-18T03:02:12.155 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2022-07-18T03:02:12.240 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2022-07-18T03:02:12.240 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2022-07-18T03:02:12.240 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-18T03:02:12.240 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-18T03:02:12.241 INFO:teuthology.orchestra.run.smithi133.stdout: debug rgw = 20 2022-07-18T03:02:12.241 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt require ssl = False 2022-07-18T03:02:12.241 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms backend = testing 2022-07-18T03:02:12.241 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-07-18T03:02:12.241 INFO:teuthology.orchestra.run.smithi133.stdout: rgw curl low speed time = 300 2022-07-18T03:02:12.283 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data log num shards = 4 2022-07-18T03:02:12.283 INFO:teuthology.orchestra.run.smithi133.stdout: rgw md log max shards = 4 2022-07-18T03:02:12.284 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync log trim interval = 0 2022-07-18T03:02:12.284 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync meta inject err probability = 0.1 2022-07-18T03:02:12.284 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync obj etag verify = True 2022-07-18T03:02:12.284 INFO:teuthology.orchestra.run.smithi133.stdout: setgroup = ceph 2022-07-18T03:02:12.285 INFO:teuthology.orchestra.run.smithi133.stdout: setuser = ceph 2022-07-18T03:02:12.285 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2022-07-18T03:02:12.285 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2022-07-18T03:02:12.285 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data notify interval msec = 0 2022-07-18T03:02:12.300 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2022-07-18T03:02:12.300 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10 2022-07-18T03:02:12.300 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2022-07-18T03:02:12.301 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 1000 2022-07-18T03:02:12.301 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2022-07-18T03:02:12.301 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10000 2022-07-18T03:02:12.303 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2022-07-18T03:02:12.303 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2022-07-18T03:02:12.369 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-18T03:02:12.369 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2022-07-18T03:02:12.370 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.370 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2022-07-18T03:02:12.371 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.371 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2022-07-18T03:02:12.390 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.413 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2022-07-18T03:02:12.413 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2022-07-18T03:02:12.414 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.414 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2022-07-18T03:02:12.414 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2022-07-18T03:02:12.414 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.415 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-18T03:02:12.415 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2022-07-18T03:02:12.415 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.415 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2022-07-18T03:02:12.416 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-18T03:02:12.416 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.436 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2022-07-18T03:02:12.437 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.437 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2022-07-18T03:02:12.437 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2022-07-18T03:02:12.437 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2022-07-18T03:02:12.437 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2022-07-18T03:02:12.438 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2022-07-18T03:02:12.438 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2022-07-18T03:02:12.438 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2022-07-18T03:02:12.438 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-18T03:02:12.439 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-18T03:02:12.439 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2022-07-18T03:02:12.439 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.474 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-07-18T03:02:12.474 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.474 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2022-07-18T03:02:12.475 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.475 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2022-07-18T03:02:12.475 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.475 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2022-07-18T03:02:12.476 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2022-07-18T03:02:12.476 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2022-07-18T03:02:12.477 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = 172.21.15.159 2022-07-18T03:02:12.480 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = c5d94c41-00fc-445f-ba02-75a44483a17a 2022-07-18T03:02:12.480 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.480 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2022-07-18T03:02:12.480 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2022-07-18T03:02:12.481 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2022-07-18T03:02:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2022-07-18T03:02:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2022-07-18T03:02:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2022-07-18T03:02:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2022-07-18T03:02:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2022-07-18T03:02:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2022-07-18T03:02:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2022-07-18T03:02:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2022-07-18T03:02:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.511 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2022-07-18T03:02:12.511 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.511 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2022-07-18T03:02:12.511 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.512 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2022-07-18T03:02:12.512 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.512 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2022-07-18T03:02:12.512 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2022-07-18T03:02:12.513 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.602 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2022-07-18T03:02:12.602 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2022-07-18T03:02:12.603 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-18T03:02:12.603 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2022-07-18T03:02:12.603 INFO:teuthology.orchestra.run.smithi159.stdout: rocksdb delete range threshold = 0 2022-07-18T03:02:12.603 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.604 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2022-07-18T03:02:12.604 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-18T03:02:12.604 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2022-07-18T03:02:12.604 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-18T03:02:12.604 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-18T03:02:12.605 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:02:12.605 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:02:12.605 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2022-07-18T03:02:12.605 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.682 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2022-07-18T03:02:12.683 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-18T03:02:12.684 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-18T03:02:12.684 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2022-07-18T03:02:12.684 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-18T03:02:12.685 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2022-07-18T03:02:12.685 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2022-07-18T03:02:12.735 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-18T03:02:12.736 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2022-07-18T03:02:12.736 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2022-07-18T03:02:12.736 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-18T03:02:12.737 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.737 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-18T03:02:12.750 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2022-07-18T03:02:12.751 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2022-07-18T03:02:12.751 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.752 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2022-07-18T03:02:12.752 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-18T03:02:12.752 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-18T03:02:12.753 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.753 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2022-07-18T03:02:12.797 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2022-07-18T03:02:12.798 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.798 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2022-07-18T03:02:12.799 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:12.799 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2022-07-18T03:02:12.799 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2022-07-18T03:02:12.800 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2022-07-18T03:02:12.814 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2022-07-18T03:02:12.833 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-18T03:02:12.833 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-18T03:02:12.834 INFO:teuthology.orchestra.run.smithi159.stdout: debug rgw = 20 2022-07-18T03:02:12.834 INFO:teuthology.orchestra.run.smithi159.stdout: rgw crypt require ssl = False 2022-07-18T03:02:12.835 INFO:teuthology.orchestra.run.smithi159.stdout: rgw crypt s3 kms backend = testing 2022-07-18T03:02:12.835 INFO:teuthology.orchestra.run.smithi159.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-07-18T03:02:12.835 INFO:teuthology.orchestra.run.smithi159.stdout: rgw curl low speed time = 300 2022-07-18T03:02:12.836 INFO:teuthology.orchestra.run.smithi159.stdout: rgw data log num shards = 4 2022-07-18T03:02:12.836 INFO:teuthology.orchestra.run.smithi159.stdout: rgw md log max shards = 4 2022-07-18T03:02:12.836 INFO:teuthology.orchestra.run.smithi159.stdout: rgw sync log trim interval = 0 2022-07-18T03:02:12.894 INFO:teuthology.orchestra.run.smithi159.stdout: rgw sync meta inject err probability = 0.1 2022-07-18T03:02:12.895 INFO:teuthology.orchestra.run.smithi159.stdout: rgw sync obj etag verify = True 2022-07-18T03:02:12.895 INFO:teuthology.orchestra.run.smithi159.stdout: setgroup = ceph 2022-07-18T03:02:12.896 INFO:teuthology.orchestra.run.smithi159.stdout: setuser = ceph 2022-07-18T03:02:12.896 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2022-07-18T03:02:12.896 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2022-07-18T03:02:12.896 INFO:teuthology.orchestra.run.smithi159.stdout: rgw data notify interval msec = 0 2022-07-18T03:02:12.897 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2022-07-18T03:02:12.897 INFO:teuthology.orchestra.run.smithi159.stdout: osd_max_omap_entries_per_request = 10 2022-07-18T03:02:12.897 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2022-07-18T03:02:12.897 INFO:teuthology.orchestra.run.smithi159.stdout: osd_max_omap_entries_per_request = 1000 2022-07-18T03:02:12.898 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2022-07-18T03:02:12.898 INFO:teuthology.orchestra.run.smithi159.stdout: osd_max_omap_entries_per_request = 10000 2022-07-18T03:02:12.899 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2022-07-18T03:02:12.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2022-07-18T03:02:12.982 INFO:tasks.ceph:Copying monmap to all nodes... 2022-07-18T03:02:12.983 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:12.983 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2022-07-18T03:02:13.049 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:13.049 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2022-07-18T03:02:13.129 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2022-07-18T03:02:13.130 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:02:13.131 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/c2.keyring 2022-07-18T03:02:13.132 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-07-18T03:02:13.185 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2022-07-18T03:02:13.186 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:13.261 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2022-07-18T03:02:13.276 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:13.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/c2.keyring 2022-07-18T03:02:13.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-07-18T03:02:13.337 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:13.338 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:13.420 INFO:tasks.ceph:Setting up mon nodes... 2022-07-18T03:02:13.421 INFO:tasks.ceph:Setting up mgr nodes... 2022-07-18T03:02:13.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2022-07-18T03:02:13.528 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2022-07-18T03:02:13.528 INFO:tasks.ceph:Setting up mds nodes... 2022-07-18T03:02:13.531 INFO:tasks.ceph_client:Setting up client nodes... 2022-07-18T03:02:13.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2022-07-18T03:02:13.596 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/c2.client.0.keyring 2022-07-18T03:02:13.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2022-07-18T03:02:13.706 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/c2.client.1.keyring 2022-07-18T03:02:13.723 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-07-18T03:02:13.723 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-07-18T03:02:13.731 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2022-07-18T03:02:13.834 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:02:13.835 INFO:tasks.ceph:role: c2.osd.0 2022-07-18T03:02:13.836 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2022-07-18T03:02:13.837 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-18T03:02:13.914 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-18T03:02:13.915 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-18T03:02:13.915 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-18T03:02:13.916 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-07-18T03:02:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-18T03:02:13.917 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2022-07-18T03:02:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-18T03:02:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-18T03:02:13.918 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-18T03:02:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-18T03:02:14.365 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2022-07-18T03:02:14.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2022-07-18T03:02:14.367 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2022-07-18T03:02:14.544 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2022-07-18T03:02:14.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2022-07-18T03:02:14.673 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:02:14.700 INFO:tasks.ceph:role: c2.osd.1 2022-07-18T03:02:14.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2022-07-18T03:02:14.718 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-18T03:02:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-18T03:02:14.771 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-18T03:02:14.771 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-18T03:02:14.772 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-07-18T03:02:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-18T03:02:14.773 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2022-07-18T03:02:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-18T03:02:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-18T03:02:14.778 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-18T03:02:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-18T03:02:15.179 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2022-07-18T03:02:15.180 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2022-07-18T03:02:15.181 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2022-07-18T03:02:15.324 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2022-07-18T03:02:15.364 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2022-07-18T03:02:15.433 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-07-18T03:02:15.433 INFO:tasks.ceph:role: c2.osd.2 2022-07-18T03:02:15.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2022-07-18T03:02:15.434 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-18T03:02:15.522 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-18T03:02:15.523 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-18T03:02:15.523 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-18T03:02:15.523 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-07-18T03:02:15.523 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-18T03:02:15.523 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2022-07-18T03:02:15.524 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-18T03:02:15.524 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-18T03:02:15.529 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-18T03:02:15.529 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-18T03:02:15.986 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2022-07-18T03:02:15.987 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2022-07-18T03:02:15.988 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2022-07-18T03:02:16.199 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2022-07-18T03:02:16.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:16.356 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:16.354+0000 7f7848b36440 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2022-07-18T03:02:16.359 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:16.356+0000 7f7848b36440 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2022-07-18T03:02:16.359 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:16.356+0000 7f7848b36440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-07-18T03:02:16.363 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:16.356+0000 7f7848b36440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-07-18T03:02:16.364 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:16.356+0000 7f7848b36440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-07-18T03:02:16.397 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:16.361+0000 7f7848b36440 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2022-07-18T03:02:18.442 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2022-07-18T03:02:18.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:18.651 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:18.630+0000 7fa63ebb6440 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2022-07-18T03:02:18.652 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:18.631+0000 7fa63ebb6440 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2022-07-18T03:02:18.652 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:18.631+0000 7fa63ebb6440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-07-18T03:02:18.653 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:18.631+0000 7fa63ebb6440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-07-18T03:02:18.653 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:18.631+0000 7fa63ebb6440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-07-18T03:02:18.654 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:18.631+0000 7fa63ebb6440 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2022-07-18T03:02:20.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2022-07-18T03:02:20.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:20.880 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:20.866+0000 7f46f797f440 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2022-07-18T03:02:20.880 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:20.867+0000 7f46f797f440 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2022-07-18T03:02:20.881 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:20.867+0000 7f46f797f440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-07-18T03:02:20.881 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:20.867+0000 7f46f797f440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-07-18T03:02:20.881 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:20.867+0000 7f46f797f440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-07-18T03:02:20.893 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:02:20.867+0000 7f46f797f440 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2022-07-18T03:02:22.949 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2022-07-18T03:02:23.067 INFO:tasks.ceph:Reading keys from all nodes... 2022-07-18T03:02:23.068 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:23.069 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2022-07-18T03:02:23.148 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:23.149 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2022-07-18T03:02:23.242 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:23.243 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2022-07-18T03:02:23.373 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:23.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2022-07-18T03:02:23.441 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:23.467 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2022-07-18T03:02:23.519 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:23.520 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2022-07-18T03:02:23.620 INFO:tasks.ceph:Adding keys to all mons... 2022-07-18T03:02:23.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/c2.keyring 2022-07-18T03:02:23.703 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2022-07-18T03:02:23.704 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA1zdRi4nsjHxAAO75baA3ESS6SsgxfpZc92Q== 2022-07-18T03:02:23.704 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2022-07-18T03:02:23.705 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA4zdRigHE5FRAA3jcZAFGFsnfWFTiJ4sro1g== 2022-07-18T03:02:23.705 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2022-07-18T03:02:23.742 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA6zdRiBnKtJRAAvHkE/PcdKy2N4L6vCi2CrQ== 2022-07-18T03:02:23.742 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2022-07-18T03:02:23.743 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA8zdRiLei+MxAA+ebRX0Ic/VItAS5+zIDBkA== 2022-07-18T03:02:23.743 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2022-07-18T03:02:23.744 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA1zdRiSbKDIxAAXI13ss1zism6R9xbJMJtqg== 2022-07-18T03:02:23.744 INFO:teuthology.orchestra.run.smithi159.stdout:[client.1] 2022-07-18T03:02:23.744 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA1zdRiIbcdKhAAW00hOLetrDBNSKDkl4N6iA== 2022-07-18T03:02:23.745 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-18T03:02:23.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-07-18T03:02:23.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-07-18T03:02:24.054 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-07-18T03:02:24.169 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-18T03:02:24.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-18T03:02:24.372 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-07-18T03:02:24.373 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2022-07-18T03:02:24.445 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2022-07-18T03:02:24.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2022-07-18T03:02:24.616 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/c2.monmap 2022-07-18T03:02:24.689 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2022-07-18T03:02:24.690 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2022-07-18T03:02:24.691 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2022-07-18T03:02:24.761 INFO:tasks.ceph.c2.mon.a:Started 2022-07-18T03:02:24.761 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2022-07-18T03:02:24.762 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2022-07-18T03:02:24.762 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2022-07-18T03:02:24.774 INFO:tasks.ceph.c2.mgr.x:Started 2022-07-18T03:02:24.775 INFO:tasks.ceph:Setting crush tunables to default 2022-07-18T03:02:24.775 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster c2 osd crush tunables default 2022-07-18T03:02:25.058 INFO:teuthology.orchestra.run.smithi159.stderr:adjusted tunables profile to default 2022-07-18T03:02:25.064 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2022-07-18T03:02:25.066 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:25.068 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2022-07-18T03:02:25.161 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:25.161 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2022-07-18T03:02:25.245 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-18T03:02:25.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2022-07-18T03:02:25.293 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.292+0000 7ff1fc6bd080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-18T03:02:25.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster c2 osd new feab4c88-8968-4170-8a56-b64d61c428ee 0 2022-07-18T03:02:25.378 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.373+0000 7ff1fc6bd080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-18T03:02:25.458 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.445+0000 7ff1fc6bd080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-18T03:02:25.547 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.545+0000 7ff1fc6bd080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-18T03:02:25.592 INFO:teuthology.orchestra.run.smithi159.stdout:0 2022-07-18T03:02:25.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster c2 osd new 6dee92ee-f433-4ddc-bee0-8b586f297c73 1 2022-07-18T03:02:25.749 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.730+0000 7ff1fc6bd080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-18T03:02:25.822 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.798+0000 7ff1fc6bd080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-18T03:02:25.880 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:25.877+0000 7ff1fc6bd080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-18T03:02:25.963 INFO:teuthology.orchestra.run.smithi159.stdout:1 2022-07-18T03:02:25.976 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster c2 osd new f8460781-c6fc-4556-9a96-2a8474a1824e 2 2022-07-18T03:02:26.224 INFO:teuthology.orchestra.run.smithi159.stdout:2 2022-07-18T03:02:26.247 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2022-07-18T03:02:26.248 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2022-07-18T03:02:26.261 INFO:tasks.ceph.c2.osd.0:Started 2022-07-18T03:02:26.268 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2022-07-18T03:02:26.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2022-07-18T03:02:26.331 INFO:tasks.ceph.c2.osd.1:Started 2022-07-18T03:02:26.333 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2022-07-18T03:02:26.334 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2022-07-18T03:02:26.339 INFO:tasks.ceph.c2.osd.2:Started 2022-07-18T03:02:26.340 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-07-18T03:02:26.582 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:26.582 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":5,"fsid":"c5d94c41-00fc-445f-ba02-75a44483a17a","created":"2022-07-18T03:02:24.876011+0000","modified":"2022-07-18T03:02:26.221606+0000","last_up_change":"0.000000","last_in_change":"2022-07-18T03:02:26.221606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"feab4c88-8968-4170-8a56-b64d61c428ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6dee92ee-f433-4ddc-bee0-8b586f297c73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f8460781-c6fc-4556-9a96-2a8474a1824e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:26.583 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:26.566+0000 7ff1fc6bd080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-18T03:02:26.597 INFO:tasks.ceph.ceph_manager.c2:[] 2022-07-18T03:02:26.598 INFO:tasks.ceph:Waiting for OSDs to come up 2022-07-18T03:02:26.649 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:26.642+0000 7ff1fc6bd080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-18T03:02:26.853 INFO:tasks.ceph.c2.osd.0.smithi159.stderr:2022-07-18T03:02:26.850+0000 7f83af762440 -1 Falling back to public interface 2022-07-18T03:02:26.901 INFO:tasks.ceph.c2.osd.1.smithi159.stderr:2022-07-18T03:02:26.895+0000 7f47b433f440 -1 Falling back to public interface 2022-07-18T03:02:26.908 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-07-18T03:02:26.953 INFO:tasks.ceph.c2.osd.2.smithi159.stderr:2022-07-18T03:02:26.940+0000 7fb3fc3ec440 -1 Falling back to public interface 2022-07-18T03:02:26.991 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:26.985+0000 7ff1fc6bd080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-18T03:02:27.106 INFO:teuthology.misc.health.smithi159.stdout: 2022-07-18T03:02:27.107 INFO:teuthology.misc.health.smithi159.stdout:{"epoch":5,"fsid":"c5d94c41-00fc-445f-ba02-75a44483a17a","created":"2022-07-18T03:02:24.876011+0000","modified":"2022-07-18T03:02:26.221606+0000","last_up_change":"0.000000","last_in_change":"2022-07-18T03:02:26.221606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"feab4c88-8968-4170-8a56-b64d61c428ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6dee92ee-f433-4ddc-bee0-8b586f297c73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f8460781-c6fc-4556-9a96-2a8474a1824e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:27.123 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-07-18T03:02:27.431 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:27.426+0000 7ff1fc6bd080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-18T03:02:27.498 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:27.497+0000 7ff1fc6bd080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-18T03:02:27.566 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:27.564+0000 7ff1fc6bd080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-18T03:02:27.720 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:27.706+0000 7ff1fc6bd080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-18T03:02:27.976 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:27.967+0000 7ff1fc6bd080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-18T03:02:28.126 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:28.096+0000 7ff1fc6bd080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-18T03:02:28.217 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:28.191+0000 7ff1fc6bd080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-18T03:02:28.297 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:28.268+0000 7ff1fc6bd080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-18T03:02:28.365 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:28.361+0000 7ff1fc6bd080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-18T03:02:29.065 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:29.059+0000 7ff1fc6bd080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-18T03:02:29.706 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:29.702+0000 7ff1fc6bd080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-18T03:02:29.928 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:29.926+0000 7ff1fc6bd080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-18T03:02:30.046 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:30.040+0000 7ff1fc6bd080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-18T03:02:30.195 INFO:tasks.ceph.c2.mgr.x.smithi159.stderr:2022-07-18T03:02:30.180+0000 7ff1fc6bd080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-18T03:02:33.436 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-07-18T03:02:33.774 INFO:teuthology.misc.health.smithi159.stdout: 2022-07-18T03:02:33.775 INFO:teuthology.misc.health.smithi159.stdout:{"epoch":5,"fsid":"c5d94c41-00fc-445f-ba02-75a44483a17a","created":"2022-07-18T03:02:24.876011+0000","modified":"2022-07-18T03:02:26.221606+0000","last_up_change":"0.000000","last_in_change":"2022-07-18T03:02:26.221606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"feab4c88-8968-4170-8a56-b64d61c428ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6dee92ee-f433-4ddc-bee0-8b586f297c73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f8460781-c6fc-4556-9a96-2a8474a1824e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:33.788 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-07-18T03:02:34.610 INFO:tasks.ceph.c2.osd.0.smithi159.stderr:2022-07-18T03:02:34.587+0000 7f83af762440 -1 osd.0 0 log_to_monitors true 2022-07-18T03:02:34.657 INFO:tasks.ceph.c2.osd.1.smithi159.stderr:2022-07-18T03:02:34.641+0000 7f47b433f440 -1 osd.1 0 log_to_monitors true 2022-07-18T03:02:34.701 INFO:tasks.ceph.c2.osd.2.smithi159.stderr:2022-07-18T03:02:34.692+0000 7fb3fc3ec440 -1 osd.2 0 log_to_monitors true 2022-07-18T03:02:37.387 INFO:tasks.ceph.c2.osd.1.smithi159.stderr:2022-07-18T03:02:37.369+0000 7f47a80ee700 -1 osd.1 0 waiting for initial osdmap 2022-07-18T03:02:37.388 INFO:tasks.ceph.c2.osd.1.smithi159.stderr:2022-07-18T03:02:37.380+0000 7f47a086d700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-18T03:02:37.389 INFO:tasks.ceph.c2.osd.2.smithi159.stderr:2022-07-18T03:02:37.380+0000 7fb3f019b700 -1 osd.2 0 waiting for initial osdmap 2022-07-18T03:02:37.400 INFO:tasks.ceph.c2.osd.2.smithi159.stderr:2022-07-18T03:02:37.390+0000 7fb3e891a700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-18T03:02:37.449 INFO:tasks.ceph.c2.osd.0.smithi159.stderr:2022-07-18T03:02:37.436+0000 7f83a3511700 -1 osd.0 0 waiting for initial osdmap 2022-07-18T03:02:37.471 INFO:tasks.ceph.c2.osd.0.smithi159.stderr:2022-07-18T03:02:37.452+0000 7f839bc90700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-18T03:02:40.127 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-07-18T03:02:40.451 INFO:teuthology.misc.health.smithi159.stdout: 2022-07-18T03:02:40.451 INFO:teuthology.misc.health.smithi159.stdout:{"epoch":9,"fsid":"c5d94c41-00fc-445f-ba02-75a44483a17a","created":"2022-07-18T03:02:24.876011+0000","modified":"2022-07-18T03:02:39.380066+0000","last_up_change":"2022-07-18T03:02:38.379332+0000","last_in_change":"2022-07-18T03:02:26.221606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"feab4c88-8968-4170-8a56-b64d61c428ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":40757},{"type":"v1","addr":"172.21.15.159:6801","nonce":40757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":40757},{"type":"v1","addr":"172.21.15.159:6803","nonce":40757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":40757},{"type":"v1","addr":"172.21.15.159:6807","nonce":40757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":40757},{"type":"v1","addr":"172.21.15.159:6805","nonce":40757}]},"public_addr":"172.21.15.159:6801/40757","cluster_addr":"172.21.15.159:6803/40757","heartbeat_back_addr":"172.21.15.159:6807/40757","heartbeat_front_addr":"172.21.15.159:6805/40757","state":["exists","up"]},{"osd":1,"uuid":"6dee92ee-f433-4ddc-bee0-8b586f297c73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":40803},{"type":"v1","addr":"172.21.15.159:6809","nonce":40803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":40803},{"type":"v1","addr":"172.21.15.159:6811","nonce":40803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":40803},{"type":"v1","addr":"172.21.15.159:6815","nonce":40803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":40803},{"type":"v1","addr":"172.21.15.159:6813","nonce":40803}]},"public_addr":"172.21.15.159:6809/40803","cluster_addr":"172.21.15.159:6811/40803","heartbeat_back_addr":"172.21.15.159:6815/40803","heartbeat_front_addr":"172.21.15.159:6813/40803","state":["exists","up"]},{"osd":2,"uuid":"f8460781-c6fc-4556-9a96-2a8474a1824e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":40841},{"type":"v1","addr":"172.21.15.159:6817","nonce":40841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":40841},{"type":"v1","addr":"172.21.15.159:6819","nonce":40841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":40841},{"type":"v1","addr":"172.21.15.159:6823","nonce":40841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":40841},{"type":"v1","addr":"172.21.15.159:6821","nonce":40841}]},"public_addr":"172.21.15.159:6817/40841","cluster_addr":"172.21.15.159:6819/40841","heartbeat_back_addr":"172.21.15.159:6823/40841","heartbeat_front_addr":"172.21.15.159:6821/40841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.548115+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.681840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.712823+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:40.478 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-07-18T03:02:40.479 INFO:tasks.ceph:Creating RBD pool 2022-07-18T03:02:40.480 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster c2 osd pool create rbd 8 2022-07-18T03:02:40.846 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'rbd' created 2022-07-18T03:02:40.863 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-07-18T03:02:41.885 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rbd' on pool 'rbd' 2022-07-18T03:02:41.886 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2022-07-18T03:02:41.887 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2022-07-18T03:02:41.887 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2022-07-18T03:02:41.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2022-07-18T03:02:41.906 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-07-18T03:02:42.323 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:42.323 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":40388},{"type":"v1","addr":"172.21.15.159:6825","nonce":40388}]},"active_addr":"172.21.15.159:6825/40388","active_change":"2022-07-18T03:02:30.195113+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3475838070}]},{"addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1588199876}]},{"addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1703619896}]}]}} 2022-07-18T03:02:42.356 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2022-07-18T03:02:42.358 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2022-07-18T03:02:42.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-07-18T03:02:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:42.748 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":11,"fsid":"c5d94c41-00fc-445f-ba02-75a44483a17a","created":"2022-07-18T03:02:24.876011+0000","modified":"2022-07-18T03:02:41.838985+0000","last_up_change":"2022-07-18T03:02:38.379332+0000","last_in_change":"2022-07-18T03:02:26.221606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-18T03:02:40.243214+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-18T03:02:40.836553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"feab4c88-8968-4170-8a56-b64d61c428ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":40757},{"type":"v1","addr":"172.21.15.159:6801","nonce":40757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":40757},{"type":"v1","addr":"172.21.15.159:6803","nonce":40757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":40757},{"type":"v1","addr":"172.21.15.159:6807","nonce":40757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":40757},{"type":"v1","addr":"172.21.15.159:6805","nonce":40757}]},"public_addr":"172.21.15.159:6801/40757","cluster_addr":"172.21.15.159:6803/40757","heartbeat_back_addr":"172.21.15.159:6807/40757","heartbeat_front_addr":"172.21.15.159:6805/40757","state":["exists","up"]},{"osd":1,"uuid":"6dee92ee-f433-4ddc-bee0-8b586f297c73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":40803},{"type":"v1","addr":"172.21.15.159:6809","nonce":40803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":40803},{"type":"v1","addr":"172.21.15.159:6811","nonce":40803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":40803},{"type":"v1","addr":"172.21.15.159:6815","nonce":40803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":40803},{"type":"v1","addr":"172.21.15.159:6813","nonce":40803}]},"public_addr":"172.21.15.159:6809/40803","cluster_addr":"172.21.15.159:6811/40803","heartbeat_back_addr":"172.21.15.159:6815/40803","heartbeat_front_addr":"172.21.15.159:6813/40803","state":["exists","up"]},{"osd":2,"uuid":"f8460781-c6fc-4556-9a96-2a8474a1824e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":40841},{"type":"v1","addr":"172.21.15.159:6817","nonce":40841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":40841},{"type":"v1","addr":"172.21.15.159:6819","nonce":40841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":40841},{"type":"v1","addr":"172.21.15.159:6823","nonce":40841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":40841},{"type":"v1","addr":"172.21.15.159:6821","nonce":40841}]},"public_addr":"172.21.15.159:6817/40841","cluster_addr":"172.21.15.159:6819/40841","heartbeat_back_addr":"172.21.15.159:6823/40841","heartbeat_front_addr":"172.21.15.159:6821/40841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.548115+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.681840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.712823+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:42.763 INFO:tasks.ceph.ceph_manager.c2:all up! 2022-07-18T03:02:42.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-07-18T03:02:43.138 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:43.139 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":12,"fsid":"c5d94c41-00fc-445f-ba02-75a44483a17a","created":"2022-07-18T03:02:24.876011+0000","modified":"2022-07-18T03:02:42.839656+0000","last_up_change":"2022-07-18T03:02:38.379332+0000","last_in_change":"2022-07-18T03:02:26.221606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-18T03:02:40.243214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-18T03:02:40.836553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"feab4c88-8968-4170-8a56-b64d61c428ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":40757},{"type":"v1","addr":"172.21.15.159:6801","nonce":40757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":40757},{"type":"v1","addr":"172.21.15.159:6803","nonce":40757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":40757},{"type":"v1","addr":"172.21.15.159:6807","nonce":40757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":40757},{"type":"v1","addr":"172.21.15.159:6805","nonce":40757}]},"public_addr":"172.21.15.159:6801/40757","cluster_addr":"172.21.15.159:6803/40757","heartbeat_back_addr":"172.21.15.159:6807/40757","heartbeat_front_addr":"172.21.15.159:6805/40757","state":["exists","up"]},{"osd":1,"uuid":"6dee92ee-f433-4ddc-bee0-8b586f297c73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":40803},{"type":"v1","addr":"172.21.15.159:6809","nonce":40803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":40803},{"type":"v1","addr":"172.21.15.159:6811","nonce":40803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":40803},{"type":"v1","addr":"172.21.15.159:6815","nonce":40803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":40803},{"type":"v1","addr":"172.21.15.159:6813","nonce":40803}]},"public_addr":"172.21.15.159:6809/40803","cluster_addr":"172.21.15.159:6811/40803","heartbeat_back_addr":"172.21.15.159:6815/40803","heartbeat_front_addr":"172.21.15.159:6813/40803","state":["exists","up"]},{"osd":2,"uuid":"f8460781-c6fc-4556-9a96-2a8474a1824e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":40841},{"type":"v1","addr":"172.21.15.159:6817","nonce":40841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":40841},{"type":"v1","addr":"172.21.15.159:6819","nonce":40841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":40841},{"type":"v1","addr":"172.21.15.159:6823","nonce":40841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":40841},{"type":"v1","addr":"172.21.15.159:6821","nonce":40841}]},"public_addr":"172.21.15.159:6817/40841","cluster_addr":"172.21.15.159:6819/40841","heartbeat_back_addr":"172.21.15.159:6823/40841","heartbeat_front_addr":"172.21.15.159:6821/40841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.548115+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.681840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-18T03:02:35.712823+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-18T03:02:43.169 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2022-07-18T03:02:43.170 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2022-07-18T03:02:43.172 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2022-07-18T03:02:43.393 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2022-07-18T03:02:43.394 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-07-18T03:02:43.396 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2022-07-18T03:02:43.396 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-07-18T03:02:43.414 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2022-07-18T03:02:43.415 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-07-18T03:02:43.759 INFO:teuthology.orchestra.run.smithi159.stdout:34359738371 2022-07-18T03:02:43.783 INFO:teuthology.orchestra.run.smithi159.stdout:34359738371 2022-07-18T03:02:43.797 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2022-07-18T03:02:43.807 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2022-07-18T03:02:43.808 INFO:teuthology.orchestra.run.smithi159.stdout:34359738371 2022-07-18T03:02:43.810 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.1 2022-07-18T03:02:44.810 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-07-18T03:02:44.811 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-07-18T03:02:44.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-07-18T03:02:45.178 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2022-07-18T03:02:45.194 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2022-07-18T03:02:45.196 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2022-07-18T03:02:45.197 DEBUG:teuthology.parallel:result is None 2022-07-18T03:02:45.203 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2022-07-18T03:02:45.213 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2022-07-18T03:02:45.226 DEBUG:teuthology.parallel:result is None 2022-07-18T03:02:45.228 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2022-07-18T03:02:45.229 DEBUG:teuthology.parallel:result is None 2022-07-18T03:02:45.229 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2022-07-18T03:02:45.272 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-07-18T03:02:45.596 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:45.597 INFO:teuthology.orchestra.run.smithi159.stderr:dumped all 2022-07-18T03:02:45.612 INFO:teuthology.orchestra.run.smithi159.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2022-07-18T03:02:44.200663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25840,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":314546960,"statfs":{"total":322122547200,"available":322096087040,"internally_reserved":0,"allocated":1482752,"data_stored":1255259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.849355+0000","last_change":"2022-07-18T03:02:41.849355+0000","last_active":"2022-07-18T03:02:41.849355+0000","last_peered":"2022-07-18T03:02:41.849355+0000","last_clean":"2022-07-18T03:02:41.849355+0000","last_became_active":"2022-07-18T03:02:41.849043+0000","last_became_peered":"2022-07-18T03:02:41.849043+0000","last_unstale":"2022-07-18T03:02:41.849355+0000","last_undegraded":"2022-07-18T03:02:41.849355+0000","last_fullsized":"2022-07-18T03:02:41.849355+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T12:40:08.180908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.849425+0000","last_change":"2022-07-18T03:02:41.849425+0000","last_active":"2022-07-18T03:02:41.849425+0000","last_peered":"2022-07-18T03:02:41.849425+0000","last_clean":"2022-07-18T03:02:41.849425+0000","last_became_active":"2022-07-18T03:02:41.849142+0000","last_became_peered":"2022-07-18T03:02:41.849142+0000","last_unstale":"2022-07-18T03:02:41.849425+0000","last_undegraded":"2022-07-18T03:02:41.849425+0000","last_fullsized":"2022-07-18T03:02:41.849425+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T10:10:37.807290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.846828+0000","last_change":"2022-07-18T03:02:41.846828+0000","last_active":"2022-07-18T03:02:41.846828+0000","last_peered":"2022-07-18T03:02:41.846828+0000","last_clean":"2022-07-18T03:02:41.846828+0000","last_became_active":"2022-07-18T03:02:41.846629+0000","last_became_peered":"2022-07-18T03:02:41.846629+0000","last_unstale":"2022-07-18T03:02:41.846828+0000","last_undegraded":"2022-07-18T03:02:41.846828+0000","last_fullsized":"2022-07-18T03:02:41.846828+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T13:15:19.701743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.851275+0000","last_change":"2022-07-18T03:02:41.846719+0000","last_active":"2022-07-18T03:02:41.851275+0000","last_peered":"2022-07-18T03:02:41.851275+0000","last_clean":"2022-07-18T03:02:41.851275+0000","last_became_active":"2022-07-18T03:02:41.846350+0000","last_became_peered":"2022-07-18T03:02:41.846350+0000","last_unstale":"2022-07-18T03:02:41.851275+0000","last_undegraded":"2022-07-18T03:02:41.851275+0000","last_fullsized":"2022-07-18T03:02:41.851275+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T14:45:13.184219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":135,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:42.293663+0000","last_change":"2022-07-18T03:02:41.849701+0000","last_active":"2022-07-18T03:02:42.293663+0000","last_peered":"2022-07-18T03:02:42.293663+0000","last_clean":"2022-07-18T03:02:42.293663+0000","last_became_active":"2022-07-18T03:02:41.848986+0000","last_became_peered":"2022-07-18T03:02:41.848986+0000","last_unstale":"2022-07-18T03:02:42.293663+0000","last_undegraded":"2022-07-18T03:02:42.293663+0000","last_fullsized":"2022-07-18T03:02:42.293663+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T11:59:26.581336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.846562+0000","last_change":"2022-07-18T03:02:41.846562+0000","last_active":"2022-07-18T03:02:41.846562+0000","last_peered":"2022-07-18T03:02:41.846562+0000","last_clean":"2022-07-18T03:02:41.846562+0000","last_became_active":"2022-07-18T03:02:41.846260+0000","last_became_peered":"2022-07-18T03:02:41.846260+0000","last_unstale":"2022-07-18T03:02:41.846562+0000","last_undegraded":"2022-07-18T03:02:41.846562+0000","last_fullsized":"2022-07-18T03:02:41.846562+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T12:49:58.024178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.852970+0000","last_change":"2022-07-18T03:02:41.851925+0000","last_active":"2022-07-18T03:02:41.852970+0000","last_peered":"2022-07-18T03:02:41.852970+0000","last_clean":"2022-07-18T03:02:41.852970+0000","last_became_active":"2022-07-18T03:02:41.851647+0000","last_became_peered":"2022-07-18T03:02:41.851647+0000","last_unstale":"2022-07-18T03:02:41.852970+0000","last_undegraded":"2022-07-18T03:02:41.852970+0000","last_fullsized":"2022-07-18T03:02:41.852970+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T06:38:33.735403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.851925+0000","last_change":"2022-07-18T03:02:41.851925+0000","last_active":"2022-07-18T03:02:41.851925+0000","last_peered":"2022-07-18T03:02:41.851925+0000","last_clean":"2022-07-18T03:02:41.851925+0000","last_became_active":"2022-07-18T03:02:41.851639+0000","last_became_peered":"2022-07-18T03:02:41.851639+0000","last_unstale":"2022-07-18T03:02:41.851925+0000","last_undegraded":"2022-07-18T03:02:41.851925+0000","last_fullsized":"2022-07-18T03:02:41.851925+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T11:52:05.106542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.847074+0000","last_change":"2022-07-18T03:02:41.847074+0000","last_active":"2022-07-18T03:02:41.847074+0000","last_peered":"2022-07-18T03:02:41.847074+0000","last_clean":"2022-07-18T03:02:41.847074+0000","last_became_active":"2022-07-18T03:02:41.846809+0000","last_became_peered":"2022-07-18T03:02:41.846809+0000","last_unstale":"2022-07-18T03:02:41.847074+0000","last_undegraded":"2022-07-18T03:02:41.847074+0000","last_fullsized":"2022-07-18T03:02:41.847074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T13:19:49.300623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":24841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-18T03:02:45.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-07-18T03:02:45.928 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:45.929 INFO:teuthology.orchestra.run.smithi159.stderr:dumped all 2022-07-18T03:02:45.936 INFO:teuthology.orchestra.run.smithi159.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2022-07-18T03:02:44.200663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25840,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":314546960,"statfs":{"total":322122547200,"available":322096087040,"internally_reserved":0,"allocated":1482752,"data_stored":1255259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.849355+0000","last_change":"2022-07-18T03:02:41.849355+0000","last_active":"2022-07-18T03:02:41.849355+0000","last_peered":"2022-07-18T03:02:41.849355+0000","last_clean":"2022-07-18T03:02:41.849355+0000","last_became_active":"2022-07-18T03:02:41.849043+0000","last_became_peered":"2022-07-18T03:02:41.849043+0000","last_unstale":"2022-07-18T03:02:41.849355+0000","last_undegraded":"2022-07-18T03:02:41.849355+0000","last_fullsized":"2022-07-18T03:02:41.849355+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T12:40:08.180908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.849425+0000","last_change":"2022-07-18T03:02:41.849425+0000","last_active":"2022-07-18T03:02:41.849425+0000","last_peered":"2022-07-18T03:02:41.849425+0000","last_clean":"2022-07-18T03:02:41.849425+0000","last_became_active":"2022-07-18T03:02:41.849142+0000","last_became_peered":"2022-07-18T03:02:41.849142+0000","last_unstale":"2022-07-18T03:02:41.849425+0000","last_undegraded":"2022-07-18T03:02:41.849425+0000","last_fullsized":"2022-07-18T03:02:41.849425+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T10:10:37.807290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.846828+0000","last_change":"2022-07-18T03:02:41.846828+0000","last_active":"2022-07-18T03:02:41.846828+0000","last_peered":"2022-07-18T03:02:41.846828+0000","last_clean":"2022-07-18T03:02:41.846828+0000","last_became_active":"2022-07-18T03:02:41.846629+0000","last_became_peered":"2022-07-18T03:02:41.846629+0000","last_unstale":"2022-07-18T03:02:41.846828+0000","last_undegraded":"2022-07-18T03:02:41.846828+0000","last_fullsized":"2022-07-18T03:02:41.846828+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T13:15:19.701743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.851275+0000","last_change":"2022-07-18T03:02:41.846719+0000","last_active":"2022-07-18T03:02:41.851275+0000","last_peered":"2022-07-18T03:02:41.851275+0000","last_clean":"2022-07-18T03:02:41.851275+0000","last_became_active":"2022-07-18T03:02:41.846350+0000","last_became_peered":"2022-07-18T03:02:41.846350+0000","last_unstale":"2022-07-18T03:02:41.851275+0000","last_undegraded":"2022-07-18T03:02:41.851275+0000","last_fullsized":"2022-07-18T03:02:41.851275+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T14:45:13.184219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":135,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:42.293663+0000","last_change":"2022-07-18T03:02:41.849701+0000","last_active":"2022-07-18T03:02:42.293663+0000","last_peered":"2022-07-18T03:02:42.293663+0000","last_clean":"2022-07-18T03:02:42.293663+0000","last_became_active":"2022-07-18T03:02:41.848986+0000","last_became_peered":"2022-07-18T03:02:41.848986+0000","last_unstale":"2022-07-18T03:02:42.293663+0000","last_undegraded":"2022-07-18T03:02:42.293663+0000","last_fullsized":"2022-07-18T03:02:42.293663+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T11:59:26.581336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.846562+0000","last_change":"2022-07-18T03:02:41.846562+0000","last_active":"2022-07-18T03:02:41.846562+0000","last_peered":"2022-07-18T03:02:41.846562+0000","last_clean":"2022-07-18T03:02:41.846562+0000","last_became_active":"2022-07-18T03:02:41.846260+0000","last_became_peered":"2022-07-18T03:02:41.846260+0000","last_unstale":"2022-07-18T03:02:41.846562+0000","last_undegraded":"2022-07-18T03:02:41.846562+0000","last_fullsized":"2022-07-18T03:02:41.846562+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T12:49:58.024178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.852970+0000","last_change":"2022-07-18T03:02:41.851925+0000","last_active":"2022-07-18T03:02:41.852970+0000","last_peered":"2022-07-18T03:02:41.852970+0000","last_clean":"2022-07-18T03:02:41.852970+0000","last_became_active":"2022-07-18T03:02:41.851647+0000","last_became_peered":"2022-07-18T03:02:41.851647+0000","last_unstale":"2022-07-18T03:02:41.852970+0000","last_undegraded":"2022-07-18T03:02:41.852970+0000","last_fullsized":"2022-07-18T03:02:41.852970+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T06:38:33.735403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.851925+0000","last_change":"2022-07-18T03:02:41.851925+0000","last_active":"2022-07-18T03:02:41.851925+0000","last_peered":"2022-07-18T03:02:41.851925+0000","last_clean":"2022-07-18T03:02:41.851925+0000","last_became_active":"2022-07-18T03:02:41.851639+0000","last_became_peered":"2022-07-18T03:02:41.851639+0000","last_unstale":"2022-07-18T03:02:41.851925+0000","last_undegraded":"2022-07-18T03:02:41.851925+0000","last_fullsized":"2022-07-18T03:02:41.851925+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T11:52:05.106542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-07-18T03:02:41.847074+0000","last_change":"2022-07-18T03:02:41.847074+0000","last_active":"2022-07-18T03:02:41.847074+0000","last_peered":"2022-07-18T03:02:41.847074+0000","last_clean":"2022-07-18T03:02:41.847074+0000","last_became_active":"2022-07-18T03:02:41.846809+0000","last_became_peered":"2022-07-18T03:02:41.846809+0000","last_unstale":"2022-07-18T03:02:41.847074+0000","last_undegraded":"2022-07-18T03:02:41.847074+0000","last_fullsized":"2022-07-18T03:02:41.847074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-18T03:02:40.836895+0000","last_clean_scrub_stamp":"2022-07-18T03:02:40.836895+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-19T13:19:49.300623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":24841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-18T03:02:45.938 INFO:tasks.ceph.ceph_manager.c2:clean! 2022-07-18T03:02:45.938 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2022-07-18T03:02:45.939 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2022-07-18T03:02:45.939 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2022-07-18T03:02:46.304 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-18T03:02:46.304 INFO:teuthology.orchestra.run.smithi159.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-18T03:02:46.312 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2022-07-18T03:02:46.312 INFO:teuthology.run_tasks:Running task rgw... 2022-07-18T03:02:46.398 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2022-07-18T03:02:46.399 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2022-07-18T03:02:46.400 INFO:tasks.rgw:Creating data pools 2022-07-18T03:02:46.400 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2022-07-18T03:02:46.401 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-07-18T03:02:47.379 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.data' created 2022-07-18T03:02:47.398 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-07-18T03:02:48.375 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-07-18T03:02:48.404 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-07-18T03:02:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.index' created 2022-07-18T03:02:49.448 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-07-18T03:02:50.412 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-07-18T03:02:50.423 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2022-07-18T03:02:50.424 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-07-18T03:02:50.772 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.data' already exists 2022-07-18T03:02:50.829 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-07-18T03:02:51.394 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-07-18T03:02:51.423 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-07-18T03:02:51.733 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.index' already exists 2022-07-18T03:02:51.742 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-07-18T03:02:52.420 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-07-18T03:02:52.427 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2022-07-18T03:02:52.428 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-07-18T03:02:52.886 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'default.rgw.buckets.data' created 2022-07-18T03:02:52.887 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-07-18T03:02:53.948 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-07-18T03:02:53.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-07-18T03:02:54.931 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'default.rgw.buckets.index' created 2022-07-18T03:02:54.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-07-18T03:02:55.908 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-07-18T03:02:55.915 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2022-07-18T03:02:55.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-07-18T03:02:56.231 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'default.rgw.buckets.data' already exists 2022-07-18T03:02:56.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-07-18T03:02:56.896 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-07-18T03:02:56.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-07-18T03:02:57.206 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'default.rgw.buckets.index' already exists 2022-07-18T03:02:57.240 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-07-18T03:02:57.900 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-07-18T03:02:57.903 DEBUG:tasks.rgw:Pools created 2022-07-18T03:02:57.904 INFO:tasks.rgw:Configuring compression type = random 2022-07-18T03:02:57.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2022-07-18T03:02:57.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2022-07-18T03:02:57.907 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2022-07-18T03:02:57.987 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:02:57.992 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:02:57.993 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:57.992+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.725 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.669+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.740 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.669+0000 7f10456a3540 20 realm 2022-07-18T03:02:59.740 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.669+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.741 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.670+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.741 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.670+0000 7f10456a3540 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-07-18T03:02:59.741 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.670+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.815 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.670+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.815 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.670+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.815 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.816 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.816 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.817 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.817 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.818 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 10 Creating default zonegroup 2022-07-18T03:02:59.830 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.671+0000 7f10456a3540 20 get_system_obj_state: rctx=0x7ffea8a265c0 obj=:.pools.avail state=0x55775b726d70 s->prefetch_data=0 2022-07-18T03:02:59.831 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.672+0000 7f10456a3540 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-07-18T03:02:59.831 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.681+0000 7f10456a3540 10 WARNING: store->list_zones() returned r=-2 2022-07-18T03:02:59.832 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.681+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.832 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.681+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.832 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.684+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.833 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.684+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.833 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.684+0000 7f10456a3540 10 could not read realm id: (2) No such file or directory 2022-07-18T03:02:59.835 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.684+0000 7f10456a3540 10 WARNING: failed to set zone as default, r=-22 2022-07-18T03:02:59.835 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.684+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.836 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.685+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.836 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.687+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.837 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:02:59.837 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.838 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:02:59.838 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 10 Using default name default 2022-07-18T03:02:59.839 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.853 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:02:59.853 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.691+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.854 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:02:59.854 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.854 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.855 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.855 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.856 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.856 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.692+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:02:59.857 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 RGWRados::pool_iterate: got zone_info.3d88cf82-6c58-450c-ac13-22234a1b15c0 2022-07-18T03:02:59.857 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 RGWRados::pool_iterate: got zonegroup_info.260c8ade-c2c6-410a-a9c4-ec2a34db1534 2022-07-18T03:02:59.858 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:02:59.858 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:02:59.858 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.859 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:02:59.859 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.700+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.860 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:02:59.860 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.861 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:02:59.861 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.862 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:02:59.863 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.863 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.864 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 zone default found 2022-07-18T03:02:59.864 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 4 Realm: () 2022-07-18T03:02:59.864 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 4 ZoneGroup: default (260c8ade-c2c6-410a-a9c4-ec2a34db1534) 2022-07-18T03:02:59.865 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 4 Zone: default (3d88cf82-6c58-450c-ac13-22234a1b15c0) 2022-07-18T03:02:59.866 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 10 cannot find current period zonegroup using local zonegroup configuration 2022-07-18T03:02:59.866 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 zonegroup default 2022-07-18T03:02:59.867 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.701+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:02:59.867 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.702+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:02:59.868 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:02:59.702+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:03:01.684 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:01.673+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:01.673+0000 7f10456a3540 20 rados->read ofs=0 len=0 2022-07-18T03:03:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:01.674+0000 7f10456a3540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:01.674+0000 7f10456a3540 20 started sync module instance, tier type = 2022-07-18T03:03:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:01.674+0000 7f10456a3540 20 started zone id=3d88cf82-6c58-450c-ac13-22234a1b15c0 (name=default) with tier type = 2022-07-18T03:03:03.719 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=0 2022-07-18T03:03:03.733 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=1 2022-07-18T03:03:03.751 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=2 2022-07-18T03:03:03.752 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=3 2022-07-18T03:03:03.752 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=4 2022-07-18T03:03:03.753 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=5 2022-07-18T03:03:03.753 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.702+0000 7f10456a3540 20 add_watcher() i=6 2022-07-18T03:03:03.755 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.703+0000 7f10456a3540 20 add_watcher() i=7 2022-07-18T03:03:03.756 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.703+0000 7f10456a3540 2 all 8 watchers are set, enabling cache 2022-07-18T03:03:03.757 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.703+0000 7f10456a3540 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-07-18T03:03:03.757 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.703+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-07-18T03:03:03.757 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.703+0000 7f10456a3540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:03.766 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-07-18T03:03:03.767 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-07-18T03:03:03.767 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:03.768 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-07-18T03:03:03.768 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-07-18T03:03:03.768 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:03.769 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-07-18T03:03:03.769 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-07-18T03:03:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-07-18T03:03:03.782 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.704+0000 7f10456a3540 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-07-18T03:03:03.782 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.705+0000 7f10456a3540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:03.782 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.708+0000 7f1009fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.708+0000 7f10456a3540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:03.708+0000 7f10456a3540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:06.705 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.679+0000 7f10456a3540 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-07-18T03:03:06.705 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.679+0000 7f10456a3540 5 note: GC not initialized 2022-07-18T03:03:06.706 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.681+0000 7f0f78ff9700 20 reqs_thread_entry: start 2022-07-18T03:03:06.706 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.681+0000 7f10456a3540 20 init_complete bucket index max shards: 11 2022-07-18T03:03:06.706 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.681+0000 7f10456a3540 10 rgw notify: Started notification manager with: 1 workers 2022-07-18T03:03:06.707 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.681+0000 7f0f6a7fc700 20 rgw notify: INFO: next queues processing will happen at: Mon Jul 18 03:03:37 2022 2022-07-18T03:03:06.708 INFO:teuthology.orchestra.run.smithi133.stderr: 2022-07-18T03:03:06.708 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.681+0000 7f0f69ffb700 20 reqs_thread_entry: start 2022-07-18T03:03:06.710 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2022-07-18T03:03:06.735 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.698+0000 7f10456a3540 20 remove_watcher() i=0 2022-07-18T03:03:06.736 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.698+0000 7f10456a3540 2 removed watcher, disabling cache 2022-07-18T03:03:06.736 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.699+0000 7f10456a3540 20 remove_watcher() i=1 2022-07-18T03:03:06.736 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.700+0000 7f10456a3540 20 remove_watcher() i=2 2022-07-18T03:03:06.747 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.701+0000 7f10456a3540 20 remove_watcher() i=3 2022-07-18T03:03:06.747 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.702+0000 7f10456a3540 20 remove_watcher() i=4 2022-07-18T03:03:06.748 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.703+0000 7f10456a3540 20 remove_watcher() i=5 2022-07-18T03:03:06.748 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.704+0000 7f10456a3540 20 remove_watcher() i=6 2022-07-18T03:03:06.749 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.705+0000 7f10456a3540 20 remove_watcher() i=7 2022-07-18T03:03:06.749 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.705+0000 7f1009fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:06.750 DEBUG:tasks.util.rgw: json result: [] 2022-07-18T03:03:06.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:06.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:06.752 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-07-18T03:03:06.804 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:06.805 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:06.811 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.809+0000 7fac71a4b540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:06.811 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.809+0000 7fac71a4b540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:06.815 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.813+0000 7fabadffb700 20 reqs_thread_entry: start 2022-07-18T03:03:06.815 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.813+0000 7fac71a4b540 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-07-18T03:03:06.816 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.813+0000 7fac71a4b540 20 rados->read ofs=0 len=0 2022-07-18T03:03:06.817 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.814+0000 7fac71a4b540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:06.817 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.814+0000 7fac71a4b540 20 rados->read ofs=0 len=0 2022-07-18T03:03:06.817 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.814+0000 7fac71a4b540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:06.825 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 RGWRados::pool_iterate: got zone_info.3d88cf82-6c58-450c-ac13-22234a1b15c0 2022-07-18T03:03:06.825 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 RGWRados::pool_iterate: got zonegroup_info.260c8ade-c2c6-410a-a9c4-ec2a34db1534 2022-07-18T03:03:06.826 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:06.826 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:06.827 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados->read ofs=0 len=0 2022-07-18T03:03:06.827 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:06.827 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados->read ofs=0 len=0 2022-07-18T03:03:06.828 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:06.828 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados->read ofs=0 len=0 2022-07-18T03:03:06.829 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:06.829 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.822+0000 7fac71a4b540 20 rados->read ofs=0 len=0 2022-07-18T03:03:06.830 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.823+0000 7fac71a4b540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:06.833 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:06.834 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "3d88cf82-6c58-450c-ac13-22234a1b15c0", 2022-07-18T03:03:06.834 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "default", 2022-07-18T03:03:06.835 INFO:teuthology.orchestra.run.smithi133.stdout: "domain_root": "default.rgw.meta:root", 2022-07-18T03:03:06.835 INFO:teuthology.orchestra.run.smithi133.stdout: "control_pool": "default.rgw.control", 2022-07-18T03:03:06.836 INFO:teuthology.orchestra.run.smithi133.stdout: "gc_pool": "default.rgw.log:gc", 2022-07-18T03:03:06.836 INFO:teuthology.orchestra.run.smithi133.stdout: "lc_pool": "default.rgw.log:lc", 2022-07-18T03:03:06.837 INFO:teuthology.orchestra.run.smithi133.stdout: "log_pool": "default.rgw.log", 2022-07-18T03:03:06.837 INFO:teuthology.orchestra.run.smithi133.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-07-18T03:03:06.837 INFO:teuthology.orchestra.run.smithi133.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-07-18T03:03:06.856 INFO:teuthology.orchestra.run.smithi133.stdout: "roles_pool": "default.rgw.meta:roles", 2022-07-18T03:03:06.857 INFO:teuthology.orchestra.run.smithi133.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-07-18T03:03:06.857 INFO:teuthology.orchestra.run.smithi133.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-07-18T03:03:06.857 INFO:teuthology.orchestra.run.smithi133.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-07-18T03:03:06.857 INFO:teuthology.orchestra.run.smithi133.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-07-18T03:03:06.858 INFO:teuthology.orchestra.run.smithi133.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-07-18T03:03:06.858 INFO:teuthology.orchestra.run.smithi133.stdout: "otp_pool": "default.rgw.otp", 2022-07-18T03:03:06.858 INFO:teuthology.orchestra.run.smithi133.stdout: "system_key": { 2022-07-18T03:03:06.858 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "", 2022-07-18T03:03:06.859 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "" 2022-07-18T03:03:06.859 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:06.859 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_pools": [ 2022-07-18T03:03:06.859 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:06.860 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "default-placement", 2022-07-18T03:03:06.860 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:03:06.860 INFO:teuthology.orchestra.run.smithi133.stdout: "index_pool": "default.rgw.buckets.index", 2022-07-18T03:03:06.860 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": { 2022-07-18T03:03:06.860 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD": { 2022-07-18T03:03:06.861 INFO:teuthology.orchestra.run.smithi133.stdout: "data_pool": "default.rgw.buckets.data", 2022-07-18T03:03:06.861 INFO:teuthology.orchestra.run.smithi133.stdout: "compression_type": "random" 2022-07-18T03:03:06.861 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:06.861 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:06.862 INFO:teuthology.orchestra.run.smithi133.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-07-18T03:03:06.877 INFO:teuthology.orchestra.run.smithi133.stdout: "index_type": 0 2022-07-18T03:03:06.877 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:06.878 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:06.878 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:06.879 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "", 2022-07-18T03:03:06.879 INFO:teuthology.orchestra.run.smithi133.stdout: "notif_pool": "default.rgw.log:notif" 2022-07-18T03:03:06.879 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:06.880 DEBUG:tasks.util.rgw: json result: {'id': '3d88cf82-6c58-450c-ac13-22234a1b15c0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-07-18T03:03:06.880 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2022-07-18T03:03:06.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2022-07-18T03:03:06.881 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2022-07-18T03:03:06.935 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:06.962 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:07.045 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.934+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.045 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.935+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.046 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.935+0000 7f466ca6d540 20 realm 2022-07-18T03:03:07.046 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.935+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.047 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.047 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-07-18T03:03:07.048 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.048 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.049 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.071 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.130 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.130 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:07.130 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.936+0000 7f466ca6d540 20 searching for the correct realm 2022-07-18T03:03:07.148 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zone_info.3d88cf82-6c58-450c-ac13-22234a1b15c0 2022-07-18T03:03:07.148 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zonegroup_info.260c8ade-c2c6-410a-a9c4-ec2a34db1534 2022-07-18T03:03:07.148 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:07.162 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:07.163 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.163 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.163 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.163 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.164 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.944+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.164 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:07.164 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.164 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.165 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.165 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.212 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.212 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.945+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.212 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.951+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zone_info.3d88cf82-6c58-450c-ac13-22234a1b15c0 2022-07-18T03:03:07.212 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.951+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zonegroup_info.260c8ade-c2c6-410a-a9c4-ec2a34db1534 2022-07-18T03:03:07.213 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.951+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:07.213 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.951+0000 7f466ca6d540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:07.213 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.951+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.213 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.214 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.214 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:07.214 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.214 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.215 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.215 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:07.215 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.952+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.215 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.215 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 zone default found 2022-07-18T03:03:07.216 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 4 Realm: () 2022-07-18T03:03:07.216 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 4 ZoneGroup: default (260c8ade-c2c6-410a-a9c4-ec2a34db1534) 2022-07-18T03:03:07.216 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 4 Zone: default (3d88cf82-6c58-450c-ac13-22234a1b15c0) 2022-07-18T03:03:07.216 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 10 cannot find current period zonegroup using local zonegroup configuration 2022-07-18T03:03:07.217 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 zonegroup default 2022-07-18T03:03:07.217 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.217 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.217 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.218 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.227 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.227 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.227 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 started sync module instance, tier type = 2022-07-18T03:03:07.228 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.953+0000 7f466ca6d540 20 started zone id=3d88cf82-6c58-450c-ac13-22234a1b15c0 (name=default) with tier type = 2022-07-18T03:03:07.228 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=0 2022-07-18T03:03:07.228 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=1 2022-07-18T03:03:07.229 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=2 2022-07-18T03:03:07.229 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=3 2022-07-18T03:03:07.229 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=4 2022-07-18T03:03:07.229 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=5 2022-07-18T03:03:07.238 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.970+0000 7f466ca6d540 20 add_watcher() i=6 2022-07-18T03:03:07.239 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.973+0000 7f466ca6d540 20 add_watcher() i=7 2022-07-18T03:03:07.239 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.973+0000 7f466ca6d540 2 all 8 watchers are set, enabling cache 2022-07-18T03:03:07.240 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.976+0000 7f45abfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:07.240 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.976+0000 7f466ca6d540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:07.240 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.976+0000 7f466ca6d540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:07.263 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.980+0000 7f466ca6d540 5 note: GC not initialized 2022-07-18T03:03:07.263 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.980+0000 7f459affd700 20 reqs_thread_entry: start 2022-07-18T03:03:07.263 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.981+0000 7f466ca6d540 20 init_complete bucket index max shards: 11 2022-07-18T03:03:07.263 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.981+0000 7f466ca6d540 10 rgw notify: Started notification manager with: 1 workers 2022-07-18T03:03:07.264 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.981+0000 7f4598ff9700 20 rgw notify: INFO: next queues processing will happen at: Mon Jul 18 03:03:37 2022 2022-07-18T03:03:07.264 INFO:teuthology.orchestra.run.smithi133.stderr: 2022-07-18T03:03:07.264 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.981+0000 7f458bfff700 20 reqs_thread_entry: start 2022-07-18T03:03:07.281 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2022-07-18T03:03:07.291 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.988+0000 7f466ca6d540 20 remove_watcher() i=0 2022-07-18T03:03:07.303 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.988+0000 7f466ca6d540 2 removed watcher, disabling cache 2022-07-18T03:03:07.303 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.991+0000 7f466ca6d540 20 remove_watcher() i=1 2022-07-18T03:03:07.304 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.992+0000 7f466ca6d540 20 remove_watcher() i=2 2022-07-18T03:03:07.304 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.995+0000 7f466ca6d540 20 remove_watcher() i=3 2022-07-18T03:03:07.304 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.995+0000 7f466ca6d540 20 remove_watcher() i=4 2022-07-18T03:03:07.304 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.996+0000 7f466ca6d540 20 remove_watcher() i=5 2022-07-18T03:03:07.305 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:06.999+0000 7f466ca6d540 20 remove_watcher() i=6 2022-07-18T03:03:07.305 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.000+0000 7f466ca6d540 20 remove_watcher() i=7 2022-07-18T03:03:07.305 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.000+0000 7f45abfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:07.305 DEBUG:tasks.util.rgw: json result: [] 2022-07-18T03:03:07.306 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:07.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:07.307 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-07-18T03:03:07.386 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:07.387 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:07.392 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.390+0000 7f87d9f64540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:07.393 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.390+0000 7f87d9f64540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:07.397 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.395+0000 7f8715ffb700 20 reqs_thread_entry: start 2022-07-18T03:03:07.398 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.395+0000 7f87d9f64540 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-07-18T03:03:07.398 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.395+0000 7f87d9f64540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.399 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.396+0000 7f87d9f64540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.399 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.396+0000 7f87d9f64540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.400 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.396+0000 7f87d9f64540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:07.416 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 RGWRados::pool_iterate: got zone_info.3d88cf82-6c58-450c-ac13-22234a1b15c0 2022-07-18T03:03:07.418 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 RGWRados::pool_iterate: got zonegroup_info.260c8ade-c2c6-410a-a9c4-ec2a34db1534 2022-07-18T03:03:07.418 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:07.418 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:07.419 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.419 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:07.420 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.420 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:07.421 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.403+0000 7f87d9f64540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.421 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.404+0000 7f87d9f64540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:07.421 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.404+0000 7f87d9f64540 20 rados->read ofs=0 len=0 2022-07-18T03:03:07.422 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:07.404+0000 7f87d9f64540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:07.441 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:07.442 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "3d88cf82-6c58-450c-ac13-22234a1b15c0", 2022-07-18T03:03:07.446 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "default", 2022-07-18T03:03:07.446 INFO:teuthology.orchestra.run.smithi133.stdout: "domain_root": "default.rgw.meta:root", 2022-07-18T03:03:07.447 INFO:teuthology.orchestra.run.smithi133.stdout: "control_pool": "default.rgw.control", 2022-07-18T03:03:07.447 INFO:teuthology.orchestra.run.smithi133.stdout: "gc_pool": "default.rgw.log:gc", 2022-07-18T03:03:07.448 INFO:teuthology.orchestra.run.smithi133.stdout: "lc_pool": "default.rgw.log:lc", 2022-07-18T03:03:07.448 INFO:teuthology.orchestra.run.smithi133.stdout: "log_pool": "default.rgw.log", 2022-07-18T03:03:07.449 INFO:teuthology.orchestra.run.smithi133.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-07-18T03:03:07.450 INFO:teuthology.orchestra.run.smithi133.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-07-18T03:03:07.450 INFO:teuthology.orchestra.run.smithi133.stdout: "roles_pool": "default.rgw.meta:roles", 2022-07-18T03:03:07.451 INFO:teuthology.orchestra.run.smithi133.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-07-18T03:03:07.452 INFO:teuthology.orchestra.run.smithi133.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-07-18T03:03:07.453 INFO:teuthology.orchestra.run.smithi133.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-07-18T03:03:07.453 INFO:teuthology.orchestra.run.smithi133.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-07-18T03:03:07.455 INFO:teuthology.orchestra.run.smithi133.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-07-18T03:03:07.456 INFO:teuthology.orchestra.run.smithi133.stdout: "otp_pool": "default.rgw.otp", 2022-07-18T03:03:07.457 INFO:teuthology.orchestra.run.smithi133.stdout: "system_key": { 2022-07-18T03:03:07.457 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "", 2022-07-18T03:03:07.458 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "" 2022-07-18T03:03:07.459 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:07.459 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_pools": [ 2022-07-18T03:03:07.460 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:07.460 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "default-placement", 2022-07-18T03:03:07.461 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:03:07.461 INFO:teuthology.orchestra.run.smithi133.stdout: "index_pool": "default.rgw.buckets.index", 2022-07-18T03:03:07.462 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": { 2022-07-18T03:03:07.462 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD": { 2022-07-18T03:03:07.463 INFO:teuthology.orchestra.run.smithi133.stdout: "data_pool": "default.rgw.buckets.data", 2022-07-18T03:03:07.463 INFO:teuthology.orchestra.run.smithi133.stdout: "compression_type": "random" 2022-07-18T03:03:07.484 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:07.484 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:07.485 INFO:teuthology.orchestra.run.smithi133.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-07-18T03:03:07.485 INFO:teuthology.orchestra.run.smithi133.stdout: "index_type": 0 2022-07-18T03:03:07.486 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:07.486 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:07.486 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:07.487 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "", 2022-07-18T03:03:07.487 INFO:teuthology.orchestra.run.smithi133.stdout: "notif_pool": "default.rgw.log:notif" 2022-07-18T03:03:07.488 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:07.489 DEBUG:tasks.util.rgw: json result: {'id': '3d88cf82-6c58-450c-ac13-22234a1b15c0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-07-18T03:03:07.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2022-07-18T03:03:07.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2022-07-18T03:03:07.491 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2022-07-18T03:03:07.596 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:07.596 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:07.597 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:07.578+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:08.876 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.870+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:08.877 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.870+0000 7fa5fafcf540 20 realm 2022-07-18T03:03:08.971 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.870+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:08.972 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.872+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:08.972 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.872+0000 7fa5fafcf540 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-07-18T03:03:08.973 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.872+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:08.973 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.873+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:08.973 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.873+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:08.974 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.873+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:08.974 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.873+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:08.975 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.874+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:08.987 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.874+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:08.988 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.874+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:08.988 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.874+0000 7fa5fafcf540 10 Creating default zonegroup 2022-07-18T03:03:08.988 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.874+0000 7fa5fafcf540 20 get_system_obj_state: rctx=0x7ffe3e9126e0 obj=:.pools.avail state=0x562bc28dbd70 s->prefetch_data=0 2022-07-18T03:03:08.989 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.874+0000 7fa5fafcf540 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-07-18T03:03:09.004 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.885+0000 7fa5fafcf540 10 WARNING: store->list_zones() returned r=-2 2022-07-18T03:03:09.004 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.885+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.005 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.885+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.005 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.887+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.006 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.887+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.006 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.887+0000 7fa5fafcf540 10 could not read realm id: (2) No such file or directory 2022-07-18T03:03:09.006 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.887+0000 7fa5fafcf540 10 WARNING: failed to set zone as default, r=-22 2022-07-18T03:03:09.007 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.887+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.007 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.888+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.008 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.890+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.009 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.890+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:09.009 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.890+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.010 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.891+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:09.010 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.891+0000 7fa5fafcf540 10 Using default name default 2022-07-18T03:03:09.011 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.891+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.011 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:09.012 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.012 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:09.013 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.013 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.036 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.037 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.038 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.892+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.038 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.896+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:09.039 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 RGWRados::pool_iterate: got zonegroup_info.842f17df-15f5-42ae-b2c4-7563ca97370d 2022-07-18T03:03:09.039 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 RGWRados::pool_iterate: got zone_info.00515f10-8170-4849-9fa7-0139bfee5d5b 2022-07-18T03:03:09.041 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:09.042 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:09.042 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.043 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:09.044 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.904+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.045 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:09.045 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.047 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:09.047 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.048 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:09.048 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.049 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.049 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 zone default found 2022-07-18T03:03:09.049 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 4 Realm: () 2022-07-18T03:03:09.050 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 4 ZoneGroup: default (842f17df-15f5-42ae-b2c4-7563ca97370d) 2022-07-18T03:03:09.050 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 4 Zone: default (00515f10-8170-4849-9fa7-0139bfee5d5b) 2022-07-18T03:03:09.052 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 10 cannot find current period zonegroup using local zonegroup configuration 2022-07-18T03:03:09.052 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 zonegroup default 2022-07-18T03:03:09.053 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.905+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:09.053 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.906+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:09.053 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:08.906+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:10.876 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:10.876+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:10.877 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:10.876+0000 7fa5fafcf540 20 rados->read ofs=0 len=0 2022-07-18T03:03:10.880 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:10.879+0000 7fa5fafcf540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:10.880 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:10.879+0000 7fa5fafcf540 20 started sync module instance, tier type = 2022-07-18T03:03:10.881 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:10.879+0000 7fa5fafcf540 20 started zone id=00515f10-8170-4849-9fa7-0139bfee5d5b (name=default) with tier type = 2022-07-18T03:03:12.921 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.919+0000 7fa5fafcf540 20 add_watcher() i=0 2022-07-18T03:03:12.921 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.920+0000 7fa5fafcf540 20 add_watcher() i=1 2022-07-18T03:03:12.922 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.920+0000 7fa5fafcf540 20 add_watcher() i=2 2022-07-18T03:03:12.922 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.920+0000 7fa5fafcf540 20 add_watcher() i=3 2022-07-18T03:03:12.923 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.920+0000 7fa5fafcf540 20 add_watcher() i=4 2022-07-18T03:03:12.923 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.920+0000 7fa5fafcf540 20 add_watcher() i=5 2022-07-18T03:03:12.923 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.920+0000 7fa5fafcf540 20 add_watcher() i=6 2022-07-18T03:03:12.929 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 add_watcher() i=7 2022-07-18T03:03:12.930 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 2 all 8 watchers are set, enabling cache 2022-07-18T03:03:12.931 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-07-18T03:03:12.932 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-07-18T03:03:12.932 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:12.933 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-07-18T03:03:12.933 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-07-18T03:03:12.933 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:12.934 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-07-18T03:03:12.935 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-07-18T03:03:12.935 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.921+0000 7fa5fafcf540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:12.937 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.922+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-07-18T03:03:12.960 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.922+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-07-18T03:03:12.961 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.922+0000 7fa5fafcf540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:12.961 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.922+0000 7fa5fafcf540 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-07-18T03:03:12.972 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.922+0000 7fa5fafcf540 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-07-18T03:03:12.972 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.923+0000 7fa5fafcf540 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-07-18T03:03:12.972 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.925+0000 7fa5befe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:12.973 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.926+0000 7fa5fafcf540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:12.973 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:12.926+0000 7fa5fafcf540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:15.896 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.880+0000 7fa5fafcf540 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-07-18T03:03:15.897 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.880+0000 7fa5fafcf540 5 note: GC not initialized 2022-07-18T03:03:15.897 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.882+0000 7fa5fafcf540 20 init_complete bucket index max shards: 11 2022-07-18T03:03:15.940 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.882+0000 7fa5fafcf540 10 rgw notify: Started notification manager with: 1 workers 2022-07-18T03:03:15.940 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.882+0000 7fa51f7fe700 20 rgw notify: INFO: next queues processing will happen at: Mon Jul 18 03:03:46 2022 2022-07-18T03:03:15.941 INFO:teuthology.orchestra.run.smithi159.stderr: 2022-07-18T03:03:15.941 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.882+0000 7fa51effd700 20 reqs_thread_entry: start 2022-07-18T03:03:15.942 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.884+0000 7fa54dffb700 20 reqs_thread_entry: start 2022-07-18T03:03:15.943 INFO:teuthology.orchestra.run.smithi159.stdout:[] 2022-07-18T03:03:15.966 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.901+0000 7fa5fafcf540 20 remove_watcher() i=0 2022-07-18T03:03:15.967 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.901+0000 7fa5fafcf540 2 removed watcher, disabling cache 2022-07-18T03:03:15.967 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.902+0000 7fa5fafcf540 20 remove_watcher() i=1 2022-07-18T03:03:15.968 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.903+0000 7fa5fafcf540 20 remove_watcher() i=2 2022-07-18T03:03:15.968 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.904+0000 7fa5fafcf540 20 remove_watcher() i=3 2022-07-18T03:03:15.969 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.904+0000 7fa5fafcf540 20 remove_watcher() i=4 2022-07-18T03:03:15.969 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.906+0000 7fa5fafcf540 20 remove_watcher() i=5 2022-07-18T03:03:15.970 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.908+0000 7fa5fafcf540 20 remove_watcher() i=6 2022-07-18T03:03:15.970 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.909+0000 7fa5fafcf540 20 remove_watcher() i=7 2022-07-18T03:03:15.971 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:15.910+0000 7fa5befe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:15.971 DEBUG:tasks.util.rgw: json result: [] 2022-07-18T03:03:15.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:15.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:15.976 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-07-18T03:03:16.048 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:16.049 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:16.049 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.046+0000 7f158ee57540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:16.081 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.046+0000 7f158ee57540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:16.092 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.050+0000 7f14caffd700 20 reqs_thread_entry: start 2022-07-18T03:03:16.093 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.050+0000 7f158ee57540 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-07-18T03:03:16.093 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.050+0000 7f158ee57540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.094 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.051+0000 7f158ee57540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.094 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.051+0000 7f158ee57540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.094 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.051+0000 7f158ee57540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:16.095 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 RGWRados::pool_iterate: got zonegroup_info.842f17df-15f5-42ae-b2c4-7563ca97370d 2022-07-18T03:03:16.095 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 RGWRados::pool_iterate: got zone_info.00515f10-8170-4849-9fa7-0139bfee5d5b 2022-07-18T03:03:16.096 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:16.096 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:16.096 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.097 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 rados_obj.operate() r=0 bl.length=873 2022-07-18T03:03:16.097 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.097 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.107 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.059+0000 7f158ee57540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.108 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.060+0000 7f158ee57540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.108 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.060+0000 7f158ee57540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.108 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.060+0000 7f158ee57540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:16.132 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:03:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "00515f10-8170-4849-9fa7-0139bfee5d5b", 2022-07-18T03:03:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "default", 2022-07-18T03:03:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "default.rgw.meta:root", 2022-07-18T03:03:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "default.rgw.control", 2022-07-18T03:03:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "default.rgw.log:gc", 2022-07-18T03:03:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "default.rgw.log:lc", 2022-07-18T03:03:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "default.rgw.log", 2022-07-18T03:03:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-07-18T03:03:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-07-18T03:03:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "default.rgw.meta:roles", 2022-07-18T03:03:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-07-18T03:03:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-07-18T03:03:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-07-18T03:03:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-07-18T03:03:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-07-18T03:03:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "default.rgw.otp", 2022-07-18T03:03:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T03:03:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "", 2022-07-18T03:03:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "" 2022-07-18T03:03:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T03:03:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T03:03:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:03:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "default.rgw.buckets.index", 2022-07-18T03:03:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T03:03:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T03:03:16.166 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "default.rgw.buckets.data", 2022-07-18T03:03:16.167 INFO:teuthology.orchestra.run.smithi159.stdout: "compression_type": "random" 2022-07-18T03:03:16.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:16.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:16.168 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-07-18T03:03:16.168 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T03:03:16.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:16.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:16.169 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:16.172 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "", 2022-07-18T03:03:16.173 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "default.rgw.log:notif" 2022-07-18T03:03:16.175 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:03:16.176 DEBUG:tasks.util.rgw: json result: {'id': '00515f10-8170-4849-9fa7-0139bfee5d5b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-07-18T03:03:16.177 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2022-07-18T03:03:16.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2022-07-18T03:03:16.178 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2022-07-18T03:03:16.239 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:16.243 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:16.244 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.231+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.244 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.249 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 realm 2022-07-18T03:03:16.250 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.250 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.251 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-07-18T03:03:16.256 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.258 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.259 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.259 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.259 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.233+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.260 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.234+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:16.261 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.234+0000 7fcc480b4540 20 searching for the correct realm 2022-07-18T03:03:16.318 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.241+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zonegroup_info.842f17df-15f5-42ae-b2c4-7563ca97370d 2022-07-18T03:03:16.334 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.241+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zone_info.00515f10-8170-4849-9fa7-0139bfee5d5b 2022-07-18T03:03:16.334 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.241+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:16.334 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.241+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:16.335 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.241+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.335 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.335 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.335 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.336 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.336 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:16.336 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.338 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.339 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.339 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.342 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.242+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.343 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.243+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.343 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zonegroup_info.842f17df-15f5-42ae-b2c4-7563ca97370d 2022-07-18T03:03:16.343 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zone_info.00515f10-8170-4849-9fa7-0139bfee5d5b 2022-07-18T03:03:16.343 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:16.344 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:16.344 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.344 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.344 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.345 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:16.345 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.249+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.345 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.345 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.345 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:16.346 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.346 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.346 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 zone default found 2022-07-18T03:03:16.376 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 4 Realm: () 2022-07-18T03:03:16.377 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 4 ZoneGroup: default (842f17df-15f5-42ae-b2c4-7563ca97370d) 2022-07-18T03:03:16.377 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 4 Zone: default (00515f10-8170-4849-9fa7-0139bfee5d5b) 2022-07-18T03:03:16.377 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 10 cannot find current period zonegroup using local zonegroup configuration 2022-07-18T03:03:16.378 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 zonegroup default 2022-07-18T03:03:16.378 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.379 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.379 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.250+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.379 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.251+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.379 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.251+0000 7fcc480b4540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.380 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.251+0000 7fcc480b4540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.380 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.251+0000 7fcc480b4540 20 started sync module instance, tier type = 2022-07-18T03:03:16.380 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.251+0000 7fcc480b4540 20 started zone id=00515f10-8170-4849-9fa7-0139bfee5d5b (name=default) with tier type = 2022-07-18T03:03:16.380 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=0 2022-07-18T03:03:16.381 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=1 2022-07-18T03:03:16.381 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=2 2022-07-18T03:03:16.381 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=3 2022-07-18T03:03:16.381 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=4 2022-07-18T03:03:16.381 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=5 2022-07-18T03:03:16.382 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.262+0000 7fcc480b4540 20 add_watcher() i=6 2022-07-18T03:03:16.382 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.263+0000 7fcc480b4540 20 add_watcher() i=7 2022-07-18T03:03:16.382 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.263+0000 7fcc480b4540 2 all 8 watchers are set, enabling cache 2022-07-18T03:03:16.386 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.264+0000 7fcc0bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:16.387 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.265+0000 7fcc480b4540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:16.387 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.265+0000 7fcc480b4540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:16.387 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.268+0000 7fcc480b4540 5 note: GC not initialized 2022-07-18T03:03:16.388 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.269+0000 7fcb7affd700 20 reqs_thread_entry: start 2022-07-18T03:03:16.388 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.269+0000 7fcc480b4540 20 init_complete bucket index max shards: 11 2022-07-18T03:03:16.388 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.269+0000 7fcc480b4540 10 rgw notify: Started notification manager with: 1 workers 2022-07-18T03:03:16.393 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.269+0000 7fcb78ff9700 20 rgw notify: INFO: next queues processing will happen at: Mon Jul 18 03:03:46 2022 2022-07-18T03:03:16.394 INFO:teuthology.orchestra.run.smithi159.stderr: 2022-07-18T03:03:16.394 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.270+0000 7fcb67fff700 20 reqs_thread_entry: start 2022-07-18T03:03:16.395 INFO:teuthology.orchestra.run.smithi159.stdout:[] 2022-07-18T03:03:16.397 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.277+0000 7fcc480b4540 20 remove_watcher() i=0 2022-07-18T03:03:16.397 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.277+0000 7fcc480b4540 2 removed watcher, disabling cache 2022-07-18T03:03:16.397 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.278+0000 7fcc480b4540 20 remove_watcher() i=1 2022-07-18T03:03:16.397 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.279+0000 7fcc480b4540 20 remove_watcher() i=2 2022-07-18T03:03:16.398 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.280+0000 7fcc480b4540 20 remove_watcher() i=3 2022-07-18T03:03:16.398 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.281+0000 7fcc480b4540 20 remove_watcher() i=4 2022-07-18T03:03:16.398 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.282+0000 7fcc480b4540 20 remove_watcher() i=5 2022-07-18T03:03:16.398 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.283+0000 7fcc480b4540 20 remove_watcher() i=6 2022-07-18T03:03:16.398 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.285+0000 7fcc480b4540 20 remove_watcher() i=7 2022-07-18T03:03:16.410 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.285+0000 7fcc0bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-07-18T03:03:16.410 DEBUG:tasks.util.rgw: json result: [] 2022-07-18T03:03:16.413 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:16.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-07-18T03:03:16.430 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-07-18T03:03:16.508 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:16.508 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:16.521 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.513+0000 7f213817c540 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-07-18T03:03:16.522 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.513+0000 7f213817c540 20 check_secure_mon_conn(): mode 1 is insecure 2022-07-18T03:03:16.527 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.516+0000 7f2073fff700 20 reqs_thread_entry: start 2022-07-18T03:03:16.528 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.516+0000 7f213817c540 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-07-18T03:03:16.528 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.516+0000 7f213817c540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.528 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.517+0000 7f213817c540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.541 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.517+0000 7f213817c540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.541 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.518+0000 7f213817c540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:16.543 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.525+0000 7f213817c540 20 RGWRados::pool_iterate: got zonegroup_info.842f17df-15f5-42ae-b2c4-7563ca97370d 2022-07-18T03:03:16.543 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.525+0000 7f213817c540 20 RGWRados::pool_iterate: got zone_info.00515f10-8170-4849-9fa7-0139bfee5d5b 2022-07-18T03:03:16.544 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.525+0000 7f213817c540 20 RGWRados::pool_iterate: got zone_names.default 2022-07-18T03:03:16.544 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.525+0000 7f213817c540 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-07-18T03:03:16.544 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.525+0000 7f213817c540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.545 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.525+0000 7f213817c540 20 rados_obj.operate() r=0 bl.length=889 2022-07-18T03:03:16.560 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.526+0000 7f213817c540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.560 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.526+0000 7f213817c540 20 rados_obj.operate() r=-2 bl.length=0 2022-07-18T03:03:16.561 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.526+0000 7f213817c540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.561 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.526+0000 7f213817c540 20 rados_obj.operate() r=0 bl.length=46 2022-07-18T03:03:16.562 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.526+0000 7f213817c540 20 rados->read ofs=0 len=0 2022-07-18T03:03:16.563 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:16.526+0000 7f213817c540 20 rados_obj.operate() r=0 bl.length=398 2022-07-18T03:03:16.671 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:03:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "00515f10-8170-4849-9fa7-0139bfee5d5b", 2022-07-18T03:03:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "default", 2022-07-18T03:03:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "default.rgw.meta:root", 2022-07-18T03:03:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "default.rgw.control", 2022-07-18T03:03:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "default.rgw.log:gc", 2022-07-18T03:03:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "default.rgw.log:lc", 2022-07-18T03:03:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "default.rgw.log", 2022-07-18T03:03:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-07-18T03:03:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-07-18T03:03:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "default.rgw.meta:roles", 2022-07-18T03:03:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-07-18T03:03:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-07-18T03:03:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-07-18T03:03:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-07-18T03:03:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-07-18T03:03:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "default.rgw.otp", 2022-07-18T03:03:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T03:03:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "", 2022-07-18T03:03:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "" 2022-07-18T03:03:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T03:03:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T03:03:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:03:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "default.rgw.buckets.index", 2022-07-18T03:03:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T03:03:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T03:03:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "default.rgw.buckets.data", 2022-07-18T03:03:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "compression_type": "random" 2022-07-18T03:03:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:16.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:16.767 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-07-18T03:03:16.767 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T03:03:16.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:16.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:16.778 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:16.787 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "", 2022-07-18T03:03:16.787 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "default.rgw.log:notif" 2022-07-18T03:03:16.788 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:03:16.789 DEBUG:tasks.util.rgw: json result: {'id': '00515f10-8170-4849-9fa7-0139bfee5d5b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-07-18T03:03:16.797 INFO:tasks.rgw:Starting rgw... 2022-07-18T03:03:16.803 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2022-07-18T03:03:16.839 INFO:tasks.rgw:Using beast as radosgw frontend 2022-07-18T03:03:16.840 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-07-18T03:03:16.902 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-07-18T03:03:16.917 INFO:tasks.rgw.c1.client.0:Started 2022-07-18T03:03:16.917 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2022-07-18T03:03:16.918 INFO:tasks.rgw:Using beast as radosgw frontend 2022-07-18T03:03:16.923 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-07-18T03:03:16.924 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-07-18T03:03:16.945 INFO:tasks.rgw.c1.client.1:Started 2022-07-18T03:03:16.945 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2022-07-18T03:03:16.975 INFO:tasks.rgw:Using beast as radosgw frontend 2022-07-18T03:03:16.975 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-07-18T03:03:16.976 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-07-18T03:03:17.003 INFO:tasks.rgw.c2.client.0:Started 2022-07-18T03:03:17.003 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2022-07-18T03:03:17.004 INFO:tasks.rgw:Using beast as radosgw frontend 2022-07-18T03:03:17.004 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-07-18T03:03:17.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-07-18T03:03:17.011 INFO:tasks.rgw.c2.client.1:Started 2022-07-18T03:03:17.013 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi133.front.sepia.ceph.com:8000/ 2022-07-18T03:03:17.019 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000/ 2022-07-18T03:03:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:17.209 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:17.214 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:17.234 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8000: Connection refused 2022-07-18T03:03:18.236 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000/ 2022-07-18T03:03:18.431 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:18.432 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:18.432 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-07-18T03:03:18.434 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2022-07-18T03:03:18.480 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi133.front.sepia.ceph.com:8001/ 2022-07-18T03:03:18.487 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001/ 2022-07-18T03:03:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:18.675 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:18.676 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:18.677 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:03:19.692 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001/ 2022-07-18T03:03:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:19.781 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:19.781 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:19.794 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:03:21.796 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001/ 2022-07-18T03:03:21.824 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:21.825 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:21.829 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:03:25.842 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001/ 2022-07-18T03:03:25.880 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:25.881 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:25.882 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-07-18T03:03:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2022-07-18T03:03:25.884 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi159.front.sepia.ceph.com:8000/ 2022-07-18T03:03:25.886 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8000/ 2022-07-18T03:03:25.948 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:25.949 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:25.950 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-07-18T03:03:25.952 INFO:teuthology.orchestra.run.smithi159.stdout:anonymous 2022-07-18T03:03:25.952 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi159.front.sepia.ceph.com:8001/ 2022-07-18T03:03:25.953 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8001/ 2022-07-18T03:03:26.016 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:26.017 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:26.019 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-07-18T03:03:26.020 INFO:teuthology.orchestra.run.smithi159.stdout:anonymous 2022-07-18T03:03:26.020 INFO:teuthology.run_tasks:Running task rgw-multisite... 2022-07-18T03:03:27.422 INFO:tasks.rgw_multisite:creating realm.. 2022-07-18T03:03:27.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:27.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:27.426 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:27.498 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:27.499 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:27.524 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:27.525 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:27.526 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-realm", 2022-07-18T03:03:27.526 INFO:teuthology.orchestra.run.smithi133.stdout: "current_period": "af1b1dd9-a63b-4529-89f0-3f46b7648654", 2022-07-18T03:03:27.527 INFO:teuthology.orchestra.run.smithi133.stdout: "epoch": 1 2022-07-18T03:03:27.527 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:27.535 INFO:tasks.rgw_multisite:creating master zonegroup.. 2022-07-18T03:03:27.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:27.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:27.542 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:27.662 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:27.663 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:27.664 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:27.613+0000 7fa1102f9540 -1 old format 2022-07-18T03:03:27.785 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:27.786 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "test-zonegroup", 2022-07-18T03:03:27.786 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zonegroup", 2022-07-18T03:03:27.787 INFO:teuthology.orchestra.run.smithi133.stdout: "api_name": "", 2022-07-18T03:03:27.787 INFO:teuthology.orchestra.run.smithi133.stdout: "is_master": true, 2022-07-18T03:03:27.788 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:03:27.788 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:03:27.789 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:27.789 INFO:teuthology.orchestra.run.smithi133.stdout: "hostnames": [], 2022-07-18T03:03:27.789 INFO:teuthology.orchestra.run.smithi133.stdout: "hostnames_s3website": [], 2022-07-18T03:03:27.948 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zone": "", 2022-07-18T03:03:27.949 INFO:teuthology.orchestra.run.smithi133.stdout: "zones": [], 2022-07-18T03:03:27.949 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_targets": [], 2022-07-18T03:03:27.950 INFO:teuthology.orchestra.run.smithi133.stdout: "default_placement": "", 2022-07-18T03:03:27.950 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:27.951 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_policy": { 2022-07-18T03:03:27.951 INFO:teuthology.orchestra.run.smithi133.stdout: "groups": [] 2022-07-18T03:03:27.952 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:27.952 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled_features": [ 2022-07-18T03:03:27.952 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:03:27.953 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:03:27.965 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:27.966 INFO:tasks.rgw_multisite:creating master zone.. 2022-07-18T03:03:27.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:27.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:27.986 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --endpoints http://smithi133.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:28.064 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:28.065 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:28.094 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:28.095 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:28.095 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone1", 2022-07-18T03:03:28.096 INFO:teuthology.orchestra.run.smithi133.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-07-18T03:03:28.096 INFO:teuthology.orchestra.run.smithi133.stdout: "control_pool": "test-zone1.rgw.control", 2022-07-18T03:03:28.096 INFO:teuthology.orchestra.run.smithi133.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-07-18T03:03:28.097 INFO:teuthology.orchestra.run.smithi133.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-07-18T03:03:28.097 INFO:teuthology.orchestra.run.smithi133.stdout: "log_pool": "test-zone1.rgw.log", 2022-07-18T03:03:28.098 INFO:teuthology.orchestra.run.smithi133.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-07-18T03:03:28.098 INFO:teuthology.orchestra.run.smithi133.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-07-18T03:03:28.099 INFO:teuthology.orchestra.run.smithi133.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-07-18T03:03:28.100 INFO:teuthology.orchestra.run.smithi133.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-07-18T03:03:28.101 INFO:teuthology.orchestra.run.smithi133.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-07-18T03:03:28.101 INFO:teuthology.orchestra.run.smithi133.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-07-18T03:03:28.102 INFO:teuthology.orchestra.run.smithi133.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-07-18T03:03:28.102 INFO:teuthology.orchestra.run.smithi133.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-07-18T03:03:28.103 INFO:teuthology.orchestra.run.smithi133.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-07-18T03:03:28.103 INFO:teuthology.orchestra.run.smithi133.stdout: "system_key": { 2022-07-18T03:03:28.104 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:03:28.104 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:03:28.104 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:28.105 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_pools": [ 2022-07-18T03:03:28.105 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:28.106 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "default-placement", 2022-07-18T03:03:28.106 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:03:28.107 INFO:teuthology.orchestra.run.smithi133.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-07-18T03:03:28.107 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": { 2022-07-18T03:03:28.108 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD": { 2022-07-18T03:03:28.108 INFO:teuthology.orchestra.run.smithi133.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2022-07-18T03:03:28.108 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:28.109 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:28.109 INFO:teuthology.orchestra.run.smithi133.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-07-18T03:03:28.110 INFO:teuthology.orchestra.run.smithi133.stdout: "index_type": 0 2022-07-18T03:03:28.122 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:28.123 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:28.123 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:28.124 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:28.124 INFO:teuthology.orchestra.run.smithi133.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-07-18T03:03:28.124 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:28.127 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2022-07-18T03:03:29.387 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'test-zone1.rgw.buckets.data' created 2022-07-18T03:03:29.387 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2022-07-18T03:03:30.378 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2022-07-18T03:03:30.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:30.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:30.409 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:30.461 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:30.461 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:30.487 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:30.487 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:30.487 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone1", 2022-07-18T03:03:30.488 INFO:teuthology.orchestra.run.smithi133.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-07-18T03:03:30.488 INFO:teuthology.orchestra.run.smithi133.stdout: "control_pool": "test-zone1.rgw.control", 2022-07-18T03:03:30.488 INFO:teuthology.orchestra.run.smithi133.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-07-18T03:03:30.488 INFO:teuthology.orchestra.run.smithi133.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-07-18T03:03:30.489 INFO:teuthology.orchestra.run.smithi133.stdout: "log_pool": "test-zone1.rgw.log", 2022-07-18T03:03:30.489 INFO:teuthology.orchestra.run.smithi133.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-07-18T03:03:30.518 INFO:teuthology.orchestra.run.smithi133.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-07-18T03:03:30.539 INFO:teuthology.orchestra.run.smithi133.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-07-18T03:03:30.539 INFO:teuthology.orchestra.run.smithi133.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-07-18T03:03:30.540 INFO:teuthology.orchestra.run.smithi133.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-07-18T03:03:30.540 INFO:teuthology.orchestra.run.smithi133.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-07-18T03:03:30.540 INFO:teuthology.orchestra.run.smithi133.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-07-18T03:03:30.540 INFO:teuthology.orchestra.run.smithi133.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-07-18T03:03:30.541 INFO:teuthology.orchestra.run.smithi133.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-07-18T03:03:30.541 INFO:teuthology.orchestra.run.smithi133.stdout: "system_key": { 2022-07-18T03:03:30.541 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:03:30.541 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:03:30.542 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:30.543 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_pools": [ 2022-07-18T03:03:30.543 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:30.543 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "default-placement", 2022-07-18T03:03:30.544 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:03:30.544 INFO:teuthology.orchestra.run.smithi133.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-07-18T03:03:30.544 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": { 2022-07-18T03:03:30.544 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD": { 2022-07-18T03:03:30.545 INFO:teuthology.orchestra.run.smithi133.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-07-18T03:03:30.545 INFO:teuthology.orchestra.run.smithi133.stdout: "compression_type": "random" 2022-07-18T03:03:30.545 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:30.545 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:30.546 INFO:teuthology.orchestra.run.smithi133.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-07-18T03:03:30.546 INFO:teuthology.orchestra.run.smithi133.stdout: "index_type": 0 2022-07-18T03:03:30.546 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:30.546 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:30.547 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:30.547 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "", 2022-07-18T03:03:30.547 INFO:teuthology.orchestra.run.smithi133.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-07-18T03:03:30.547 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:30.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:30.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:30.553 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:30.607 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:30.607 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:30.636 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:03:30.617+0000 7f685ffdd540 0 period (af1b1dd9-a63b-4529-89f0-3f46b7648654 does not have zone 9b07f48a-ef73-4ad5-94be-e184c9a423d4 configured 2022-07-18T03:03:37.448 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:37.448 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:03:37.448 INFO:teuthology.orchestra.run.smithi133.stdout: "epoch": 1, 2022-07-18T03:03:37.449 INFO:teuthology.orchestra.run.smithi133.stdout: "predecessor_uuid": "af1b1dd9-a63b-4529-89f0-3f46b7648654", 2022-07-18T03:03:37.449 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": [], 2022-07-18T03:03:37.449 INFO:teuthology.orchestra.run.smithi133.stdout: "period_map": { 2022-07-18T03:03:37.449 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:03:37.449 INFO:teuthology.orchestra.run.smithi133.stdout: "zonegroups": [ 2022-07-18T03:03:37.450 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:37.451 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "test-zonegroup", 2022-07-18T03:03:37.451 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zonegroup", 2022-07-18T03:03:37.452 INFO:teuthology.orchestra.run.smithi133.stdout: "api_name": "", 2022-07-18T03:03:37.452 INFO:teuthology.orchestra.run.smithi133.stdout: "is_master": true, 2022-07-18T03:03:37.452 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:03:37.452 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:03:37.453 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:37.453 INFO:teuthology.orchestra.run.smithi133.stdout: "hostnames": [], 2022-07-18T03:03:37.453 INFO:teuthology.orchestra.run.smithi133.stdout: "hostnames_s3website": [], 2022-07-18T03:03:37.454 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:37.454 INFO:teuthology.orchestra.run.smithi133.stdout: "zones": [ 2022-07-18T03:03:37.454 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:37.454 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:37.455 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone1", 2022-07-18T03:03:37.455 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:03:37.455 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:03:37.455 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:37.462 INFO:teuthology.orchestra.run.smithi133.stdout: "log_meta": false, 2022-07-18T03:03:37.463 INFO:teuthology.orchestra.run.smithi133.stdout: "log_data": false, 2022-07-18T03:03:37.463 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:03:37.463 INFO:teuthology.orchestra.run.smithi133.stdout: "read_only": false, 2022-07-18T03:03:37.463 INFO:teuthology.orchestra.run.smithi133.stdout: "tier_type": "", 2022-07-18T03:03:37.464 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from_all": true, 2022-07-18T03:03:37.464 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from": [], 2022-07-18T03:03:37.464 INFO:teuthology.orchestra.run.smithi133.stdout: "redirect_zone": "", 2022-07-18T03:03:37.464 INFO:teuthology.orchestra.run.smithi133.stdout: "supported_features": [ 2022-07-18T03:03:37.464 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:03:37.465 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:03:37.465 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:37.465 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:37.465 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_targets": [ 2022-07-18T03:03:37.466 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:37.467 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "default-placement", 2022-07-18T03:03:37.468 INFO:teuthology.orchestra.run.smithi133.stdout: "tags": [], 2022-07-18T03:03:37.468 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": [ 2022-07-18T03:03:37.468 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD" 2022-07-18T03:03:37.468 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:03:37.468 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:37.469 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:37.469 INFO:teuthology.orchestra.run.smithi133.stdout: "default_placement": "default-placement", 2022-07-18T03:03:37.469 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:37.469 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_policy": { 2022-07-18T03:03:37.470 INFO:teuthology.orchestra.run.smithi133.stdout: "groups": [] 2022-07-18T03:03:37.470 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.470 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled_features": [ 2022-07-18T03:03:37.470 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:03:37.471 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:03:37.471 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:37.471 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:37.471 INFO:teuthology.orchestra.run.smithi133.stdout: "short_zone_ids": [ 2022-07-18T03:03:37.471 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:37.472 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:37.472 INFO:teuthology.orchestra.run.smithi133.stdout: "val": 3166817753 2022-07-18T03:03:37.472 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:37.472 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:03:37.473 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.473 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zonegroup": "test-zonegroup", 2022-07-18T03:03:37.473 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:37.473 INFO:teuthology.orchestra.run.smithi133.stdout: "period_config": { 2022-07-18T03:03:37.474 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_quota": { 2022-07-18T03:03:37.474 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:03:37.475 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:03:37.475 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:03:37.475 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:03:37.475 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:03:37.489 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.489 INFO:teuthology.orchestra.run.smithi133.stdout: "user_quota": { 2022-07-18T03:03:37.489 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:03:37.490 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:03:37.490 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:03:37.490 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:03:37.490 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:03:37.491 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.491 INFO:teuthology.orchestra.run.smithi133.stdout: "user_ratelimit": { 2022-07-18T03:03:37.491 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_ops": 0, 2022-07-18T03:03:37.491 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_ops": 0, 2022-07-18T03:03:37.491 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_bytes": 0, 2022-07-18T03:03:37.492 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_bytes": 0, 2022-07-18T03:03:37.492 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false 2022-07-18T03:03:37.492 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.492 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_ratelimit": { 2022-07-18T03:03:37.493 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_ops": 0, 2022-07-18T03:03:37.493 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_ops": 0, 2022-07-18T03:03:37.493 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_bytes": 0, 2022-07-18T03:03:37.493 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_bytes": 0, 2022-07-18T03:03:37.494 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false 2022-07-18T03:03:37.498 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.499 INFO:teuthology.orchestra.run.smithi133.stdout: "anonymous_ratelimit": { 2022-07-18T03:03:37.499 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_ops": 0, 2022-07-18T03:03:37.499 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_ops": 0, 2022-07-18T03:03:37.499 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_bytes": 0, 2022-07-18T03:03:37.500 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_bytes": 0, 2022-07-18T03:03:37.500 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false 2022-07-18T03:03:37.500 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:37.500 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:37.501 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:37.501 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_name": "test-realm", 2022-07-18T03:03:37.501 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:03:37.501 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:37.503 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-07-18T03:03:37.504 INFO:teuthology.orchestra.run:waiting for 300 2022-07-18T03:03:37.508 INFO:tasks.rgw.c1.client.0.smithi133.stdout:2022-07-18T03:03:37.507+0000 7f471af6a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 46181) UID: 0 2022-07-18T03:03:37.509 INFO:tasks.rgw.c1.client.0.smithi133.stdout:2022-07-18T03:03:37.507+0000 7f472bae8600 -1 shutting down 2022-07-18T03:03:37.551 INFO:tasks.rgw.c1.client.0:Stopped 2022-07-18T03:03:37.552 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-07-18T03:03:37.552 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-07-18T03:03:37.595 INFO:tasks.rgw.c1.client.0:Started 2022-07-18T03:03:37.595 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000 2022-07-18T03:03:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:37.626 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:37.626 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:37.626 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8000: Connection refused 2022-07-18T03:03:38.634 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000 2022-07-18T03:03:38.684 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:38.684 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:38.685 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-07-18T03:03:38.688 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2022-07-18T03:03:38.689 INFO:tasks.rgw_multisite:creating admin user.. 2022-07-18T03:03:38.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:38.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:38.693 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:38.790 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:38.791 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:38.844 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:03:38.845 INFO:teuthology.orchestra.run.smithi133.stdout: "user_id": "realm-admin", 2022-07-18T03:03:38.858 INFO:teuthology.orchestra.run.smithi133.stdout: "display_name": "Realm Admin", 2022-07-18T03:03:38.858 INFO:teuthology.orchestra.run.smithi133.stdout: "email": "", 2022-07-18T03:03:38.859 INFO:teuthology.orchestra.run.smithi133.stdout: "suspended": 0, 2022-07-18T03:03:38.860 INFO:teuthology.orchestra.run.smithi133.stdout: "max_buckets": 1000, 2022-07-18T03:03:38.860 INFO:teuthology.orchestra.run.smithi133.stdout: "subusers": [], 2022-07-18T03:03:38.861 INFO:teuthology.orchestra.run.smithi133.stdout: "keys": [ 2022-07-18T03:03:38.861 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:03:38.891 INFO:teuthology.orchestra.run.smithi133.stdout: "user": "realm-admin", 2022-07-18T03:03:38.892 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:03:38.892 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:03:38.893 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:03:38.893 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:03:38.894 INFO:teuthology.orchestra.run.smithi133.stdout: "swift_keys": [], 2022-07-18T03:03:38.894 INFO:teuthology.orchestra.run.smithi133.stdout: "caps": [], 2022-07-18T03:03:38.895 INFO:teuthology.orchestra.run.smithi133.stdout: "op_mask": "read, write, delete", 2022-07-18T03:03:38.903 INFO:teuthology.orchestra.run.smithi133.stdout: "system": true, 2022-07-18T03:03:38.904 INFO:teuthology.orchestra.run.smithi133.stdout: "default_placement": "", 2022-07-18T03:03:38.904 INFO:teuthology.orchestra.run.smithi133.stdout: "default_storage_class": "", 2022-07-18T03:03:38.905 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_tags": [], 2022-07-18T03:03:38.905 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_quota": { 2022-07-18T03:03:38.906 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:03:38.906 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:03:38.907 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:03:38.907 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:03:38.908 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:03:38.908 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:38.909 INFO:teuthology.orchestra.run.smithi133.stdout: "user_quota": { 2022-07-18T03:03:38.909 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:03:38.910 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:03:38.910 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:03:38.911 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:03:38.911 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:03:38.911 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:03:38.912 INFO:teuthology.orchestra.run.smithi133.stdout: "temp_url_keys": [], 2022-07-18T03:03:38.912 INFO:teuthology.orchestra.run.smithi133.stdout: "type": "rgw", 2022-07-18T03:03:38.913 INFO:teuthology.orchestra.run.smithi133.stdout: "mfa_ids": [] 2022-07-18T03:03:38.913 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:03:38.923 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:03:38.925 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-07-18T03:03:38.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi133.front.sepia.ceph.com:8000', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:38.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi133.front.sepia.ceph.com:8000', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:38.930 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi133.front.sepia.ceph.com:8000 --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:38.992 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:38.994 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:39.038 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:03:39.093 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:39.101 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-realm", 2022-07-18T03:03:39.110 INFO:teuthology.orchestra.run.smithi159.stdout: "current_period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:03:39.111 INFO:teuthology.orchestra.run.smithi159.stdout: "epoch": 2 2022-07-18T03:03:39.111 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:03:39.122 INFO:tasks.rgw_multisite:creating zone.. 2022-07-18T03:03:39.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi159.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:39.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi159.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:39.124 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --endpoints http://smithi159.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:39.191 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:39.191 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:39.263 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:39.215+0000 7efc82c43540 0 failed reading obj info from .rgw.root:zone_info.9b07f48a-ef73-4ad5-94be-e184c9a423d4: (2) No such file or directory 2022-07-18T03:03:39.272 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:39.215+0000 7efc82c43540 0 WARNING: could not read zone params for zone id=9b07f48a-ef73-4ad5-94be-e184c9a423d4 name=test-zone1 2022-07-18T03:03:39.275 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:03:39.275 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:03:39.276 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone2", 2022-07-18T03:03:39.276 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-07-18T03:03:39.277 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "test-zone2.rgw.control", 2022-07-18T03:03:39.277 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-07-18T03:03:39.279 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-07-18T03:03:39.279 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "test-zone2.rgw.log", 2022-07-18T03:03:39.280 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-07-18T03:03:39.280 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-07-18T03:03:39.280 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-07-18T03:03:39.280 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-07-18T03:03:39.281 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-07-18T03:03:39.281 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-07-18T03:03:39.281 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-07-18T03:03:39.281 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-07-18T03:03:39.281 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-07-18T03:03:39.283 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T03:03:39.283 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:03:39.284 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:03:39.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:39.284 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T03:03:39.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:39.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T03:03:39.287 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:03:39.288 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-07-18T03:03:39.288 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T03:03:39.289 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T03:03:39.289 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2022-07-18T03:03:39.289 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:39.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:39.289 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-07-18T03:03:39.290 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T03:03:39.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:39.292 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:39.294 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:39.295 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:39.295 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-07-18T03:03:39.295 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:03:39.296 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2022-07-18T03:03:39.869 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'test-zone2.rgw.buckets.data' created 2022-07-18T03:03:39.879 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2022-07-18T03:03:40.892 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2022-07-18T03:03:40.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:40.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:40.913 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:41.007 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:41.008 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:41.034 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:03:41.035 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:03:41.035 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone2", 2022-07-18T03:03:41.036 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-07-18T03:03:41.036 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "test-zone2.rgw.control", 2022-07-18T03:03:41.037 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-07-18T03:03:41.037 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-07-18T03:03:41.038 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "test-zone2.rgw.log", 2022-07-18T03:03:41.038 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-07-18T03:03:41.039 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-07-18T03:03:41.039 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-07-18T03:03:41.040 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-07-18T03:03:41.040 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-07-18T03:03:41.041 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-07-18T03:03:41.043 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-07-18T03:03:41.044 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-07-18T03:03:41.045 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-07-18T03:03:41.045 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T03:03:41.046 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:03:41.046 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:03:41.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:41.047 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T03:03:41.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:41.048 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T03:03:41.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:03:41.049 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-07-18T03:03:41.050 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T03:03:41.050 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T03:03:41.051 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-07-18T03:03:41.058 INFO:teuthology.orchestra.run.smithi159.stdout: "compression_type": "random" 2022-07-18T03:03:41.058 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:41.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:41.059 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-07-18T03:03:41.059 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T03:03:41.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:41.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:41.091 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:41.091 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "", 2022-07-18T03:03:41.091 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-07-18T03:03:41.092 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:03:41.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:41.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:03:41.093 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:03:41.176 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:03:41.176 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:03:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:03:41.166+0000 7f8a89727540 0 period (c2173771-3b17-40d3-8c8b-0df64165bc30 does not have zone d93426b5-d0cd-461c-b03d-6cd238c57609 configured 2022-07-18T03:03:47.962 INFO:teuthology.orchestra.run.smithi159.stderr:Sending period to new master zone 9b07f48a-ef73-4ad5-94be-e184c9a423d4 2022-07-18T03:03:47.974 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:03:47.974 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:03:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "epoch": 2, 2022-07-18T03:03:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "predecessor_uuid": "af1b1dd9-a63b-4529-89f0-3f46b7648654", 2022-07-18T03:03:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": [], 2022-07-18T03:03:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "period_map": { 2022-07-18T03:03:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:03:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: "zonegroups": [ 2022-07-18T03:03:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "test-zonegroup", 2022-07-18T03:03:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zonegroup", 2022-07-18T03:03:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: "api_name": "", 2022-07-18T03:03:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: "is_master": true, 2022-07-18T03:03:47.978 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:03:47.980 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:03:47.980 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "hostnames": [], 2022-07-18T03:03:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "hostnames_s3website": [], 2022-07-18T03:03:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: "zones": [ 2022-07-18T03:03:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone1", 2022-07-18T03:03:47.983 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:03:47.983 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:03:47.985 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:47.985 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T03:03:47.986 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T03:03:47.987 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:03:47.987 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T03:03:47.987 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T03:03:47.987 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T03:03:47.988 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T03:03:47.988 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T03:03:47.988 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T03:03:47.988 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:03:47.989 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:03:47.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:47.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:47.989 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:03:47.990 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone2", 2022-07-18T03:03:47.990 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:03:47.990 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi159.front.sepia.ceph.com:8000" 2022-07-18T03:03:47.990 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:47.991 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T03:03:47.991 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T03:03:47.991 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:03:47.991 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T03:03:47.992 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T03:03:47.993 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T03:03:47.993 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T03:03:48.006 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T03:03:48.007 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T03:03:48.007 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:03:48.008 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:03:48.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:48.009 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:48.081 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_targets": [ 2022-07-18T03:03:48.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:48.083 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "default-placement", 2022-07-18T03:03:48.083 INFO:teuthology.orchestra.run.smithi159.stdout: "tags": [], 2022-07-18T03:03:48.084 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": [ 2022-07-18T03:03:48.084 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD" 2022-07-18T03:03:48.084 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:03:48.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:48.085 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:48.085 INFO:teuthology.orchestra.run.smithi159.stdout: "default_placement": "default-placement", 2022-07-18T03:03:48.085 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:48.085 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_policy": { 2022-07-18T03:03:48.085 INFO:teuthology.orchestra.run.smithi159.stdout: "groups": [] 2022-07-18T03:03:48.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.086 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled_features": [ 2022-07-18T03:03:48.086 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:03:48.086 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:03:48.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:48.087 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:03:48.087 INFO:teuthology.orchestra.run.smithi159.stdout: "short_zone_ids": [ 2022-07-18T03:03:48.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:48.087 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:48.088 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 3166817753 2022-07-18T03:03:48.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:03:48.088 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:03:48.089 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 1682003393 2022-07-18T03:03:48.089 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:48.089 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:03:48.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.089 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zonegroup": "test-zonegroup", 2022-07-18T03:03:48.105 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:03:48.105 INFO:teuthology.orchestra.run.smithi159.stdout: "period_config": { 2022-07-18T03:03:48.105 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_quota": { 2022-07-18T03:03:48.106 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false, 2022-07-18T03:03:48.109 INFO:teuthology.orchestra.run.smithi159.stdout: "check_on_raw": false, 2022-07-18T03:03:48.110 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size": -1, 2022-07-18T03:03:48.110 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size_kb": 0, 2022-07-18T03:03:48.110 INFO:teuthology.orchestra.run.smithi159.stdout: "max_objects": -1 2022-07-18T03:03:48.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.111 INFO:teuthology.orchestra.run.smithi159.stdout: "user_quota": { 2022-07-18T03:03:48.111 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false, 2022-07-18T03:03:48.112 INFO:teuthology.orchestra.run.smithi159.stdout: "check_on_raw": false, 2022-07-18T03:03:48.112 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size": -1, 2022-07-18T03:03:48.112 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size_kb": 0, 2022-07-18T03:03:48.112 INFO:teuthology.orchestra.run.smithi159.stdout: "max_objects": -1 2022-07-18T03:03:48.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.113 INFO:teuthology.orchestra.run.smithi159.stdout: "user_ratelimit": { 2022-07-18T03:03:48.113 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T03:03:48.113 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T03:03:48.113 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T03:03:48.114 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T03:03:48.114 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T03:03:48.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.114 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_ratelimit": { 2022-07-18T03:03:48.115 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T03:03:48.115 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T03:03:48.115 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T03:03:48.115 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T03:03:48.115 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T03:03:48.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.117 INFO:teuthology.orchestra.run.smithi159.stdout: "anonymous_ratelimit": { 2022-07-18T03:03:48.117 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T03:03:48.118 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T03:03:48.118 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T03:03:48.118 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T03:03:48.119 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T03:03:48.119 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:03:48.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:03:48.119 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:03:48.119 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_name": "test-realm", 2022-07-18T03:03:48.120 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:03:48.120 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:03:48.121 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-07-18T03:03:48.121 INFO:teuthology.orchestra.run:waiting for 300 2022-07-18T03:03:48.161 INFO:tasks.rgw.c2.client.0.smithi159.stdout:2022-07-18T03:03:48.158+0000 7f4f4951c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 46055) UID: 0 2022-07-18T03:03:48.161 INFO:tasks.rgw.c2.client.0.smithi159.stdout:2022-07-18T03:03:48.158+0000 7f4f5a09a600 -1 shutting down 2022-07-18T03:03:51.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-07-18T03:03:54.249 INFO:tasks.rgw.c2.client.0:Stopped 2022-07-18T03:03:54.250 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-07-18T03:03:54.250 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-07-18T03:03:54.259 INFO:tasks.rgw.c2.client.0:Started 2022-07-18T03:03:54.260 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8000 2022-07-18T03:03:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:54.293 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:54.295 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:54.296 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi159.front.sepia.ceph.com port 8000: Connection refused 2022-07-18T03:03:55.298 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8000 2022-07-18T03:03:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:55.335 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:55.335 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:55.336 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi159.front.sepia.ceph.com port 8000: Connection refused 2022-07-18T03:03:57.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-07-18T03:03:57.389 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8000 2022-07-18T03:03:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:03:57.645 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:03:57.646 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:03:57.646 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi159.front.sepia.ceph.com port 8000: Connection refused 2022-07-18T03:04:01.651 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8000 2022-07-18T03:04:01.695 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:01.695 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:01.696 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-07-18T03:04:01.697 INFO:teuthology.orchestra.run.smithi159.stdout:anonymous 2022-07-18T03:04:01.698 INFO:tasks.rgw_multisite:creating zone.. 2022-07-18T03:04:01.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:01.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:01.699 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --endpoints http://smithi133.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:01.770 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:01.771 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:01.795 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:04:01.790+0000 7fc48e617540 0 failed reading obj info from .rgw.root:zone_info.d93426b5-d0cd-461c-b03d-6cd238c57609: (2) No such file or directory 2022-07-18T03:04:01.796 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:04:01.790+0000 7fc48e617540 0 WARNING: could not read zone params for zone id=d93426b5-d0cd-461c-b03d-6cd238c57609 name=test-zone2 2022-07-18T03:04:01.797 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:04:01.847 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T03:04:01.847 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone3", 2022-07-18T03:04:01.848 INFO:teuthology.orchestra.run.smithi133.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-07-18T03:04:01.848 INFO:teuthology.orchestra.run.smithi133.stdout: "control_pool": "test-zone3.rgw.control", 2022-07-18T03:04:01.848 INFO:teuthology.orchestra.run.smithi133.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-07-18T03:04:01.849 INFO:teuthology.orchestra.run.smithi133.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-07-18T03:04:01.849 INFO:teuthology.orchestra.run.smithi133.stdout: "log_pool": "test-zone3.rgw.log", 2022-07-18T03:04:01.935 INFO:teuthology.orchestra.run.smithi133.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-07-18T03:04:01.935 INFO:teuthology.orchestra.run.smithi133.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-07-18T03:04:01.936 INFO:teuthology.orchestra.run.smithi133.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-07-18T03:04:01.936 INFO:teuthology.orchestra.run.smithi133.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-07-18T03:04:01.937 INFO:teuthology.orchestra.run.smithi133.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-07-18T03:04:01.937 INFO:teuthology.orchestra.run.smithi133.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-07-18T03:04:01.987 INFO:teuthology.orchestra.run.smithi133.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-07-18T03:04:01.989 INFO:teuthology.orchestra.run.smithi133.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-07-18T03:04:01.989 INFO:teuthology.orchestra.run.smithi133.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-07-18T03:04:02.047 INFO:teuthology.orchestra.run.smithi133.stdout: "system_key": { 2022-07-18T03:04:02.047 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:04:02.048 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:04:02.049 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:02.055 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_pools": [ 2022-07-18T03:04:02.055 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:02.056 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "default-placement", 2022-07-18T03:04:02.152 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:04:02.152 INFO:teuthology.orchestra.run.smithi133.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-07-18T03:04:02.153 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": { 2022-07-18T03:04:02.154 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD": { 2022-07-18T03:04:02.181 INFO:teuthology.orchestra.run.smithi133.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2022-07-18T03:04:02.181 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:02.182 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:02.182 INFO:teuthology.orchestra.run.smithi133.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-07-18T03:04:02.183 INFO:teuthology.orchestra.run.smithi133.stdout: "index_type": 0 2022-07-18T03:04:02.183 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:02.184 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:02.184 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:02.185 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:02.185 INFO:teuthology.orchestra.run.smithi133.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-07-18T03:04:02.277 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:04:02.284 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2022-07-18T03:04:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'test-zone3.rgw.buckets.data' created 2022-07-18T03:04:03.404 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2022-07-18T03:04:04.417 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2022-07-18T03:04:04.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:04.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:04.439 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:04.563 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:04.563 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:04.572 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:04:04.573 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T03:04:04.573 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone3", 2022-07-18T03:04:04.574 INFO:teuthology.orchestra.run.smithi133.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-07-18T03:04:04.574 INFO:teuthology.orchestra.run.smithi133.stdout: "control_pool": "test-zone3.rgw.control", 2022-07-18T03:04:04.574 INFO:teuthology.orchestra.run.smithi133.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-07-18T03:04:04.574 INFO:teuthology.orchestra.run.smithi133.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-07-18T03:04:04.575 INFO:teuthology.orchestra.run.smithi133.stdout: "log_pool": "test-zone3.rgw.log", 2022-07-18T03:04:04.575 INFO:teuthology.orchestra.run.smithi133.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-07-18T03:04:04.575 INFO:teuthology.orchestra.run.smithi133.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-07-18T03:04:04.575 INFO:teuthology.orchestra.run.smithi133.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-07-18T03:04:04.576 INFO:teuthology.orchestra.run.smithi133.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-07-18T03:04:04.576 INFO:teuthology.orchestra.run.smithi133.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-07-18T03:04:04.576 INFO:teuthology.orchestra.run.smithi133.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-07-18T03:04:04.576 INFO:teuthology.orchestra.run.smithi133.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-07-18T03:04:04.577 INFO:teuthology.orchestra.run.smithi133.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-07-18T03:04:04.577 INFO:teuthology.orchestra.run.smithi133.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-07-18T03:04:04.577 INFO:teuthology.orchestra.run.smithi133.stdout: "system_key": { 2022-07-18T03:04:04.577 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:04:04.578 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:04:04.578 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:04.578 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_pools": [ 2022-07-18T03:04:04.579 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:04.579 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "default-placement", 2022-07-18T03:04:04.579 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:04:04.580 INFO:teuthology.orchestra.run.smithi133.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-07-18T03:04:04.580 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": { 2022-07-18T03:04:04.580 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD": { 2022-07-18T03:04:04.580 INFO:teuthology.orchestra.run.smithi133.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2022-07-18T03:04:04.581 INFO:teuthology.orchestra.run.smithi133.stdout: "compression_type": "random" 2022-07-18T03:04:04.581 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:04.581 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:04.581 INFO:teuthology.orchestra.run.smithi133.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-07-18T03:04:04.582 INFO:teuthology.orchestra.run.smithi133.stdout: "index_type": 0 2022-07-18T03:04:04.582 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:04.582 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:04.582 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:04.582 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "", 2022-07-18T03:04:04.583 INFO:teuthology.orchestra.run.smithi133.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-07-18T03:04:04.583 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:04:04.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:04.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:04.586 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:04.646 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:04.647 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:04.647 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T03:04:04.644+0000 7fa3cd981540 0 period (c2173771-3b17-40d3-8c8b-0df64165bc30 does not have zone ea63e6cf-483a-453e-81a7-bf1983b4a211 configured 2022-07-18T03:04:11.448 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 9b07f48a-ef73-4ad5-94be-e184c9a423d4 2022-07-18T03:04:11.482 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:04:11.483 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:11.483 INFO:teuthology.orchestra.run.smithi133.stdout: "epoch": 3, 2022-07-18T03:04:11.483 INFO:teuthology.orchestra.run.smithi133.stdout: "predecessor_uuid": "af1b1dd9-a63b-4529-89f0-3f46b7648654", 2022-07-18T03:04:11.483 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": [], 2022-07-18T03:04:11.484 INFO:teuthology.orchestra.run.smithi133.stdout: "period_map": { 2022-07-18T03:04:11.484 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:11.484 INFO:teuthology.orchestra.run.smithi133.stdout: "zonegroups": [ 2022-07-18T03:04:11.484 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.485 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "test-zonegroup", 2022-07-18T03:04:11.485 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zonegroup", 2022-07-18T03:04:11.485 INFO:teuthology.orchestra.run.smithi133.stdout: "api_name": "", 2022-07-18T03:04:11.485 INFO:teuthology.orchestra.run.smithi133.stdout: "is_master": true, 2022-07-18T03:04:11.486 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:04:11.486 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:04:11.486 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.486 INFO:teuthology.orchestra.run.smithi133.stdout: "hostnames": [], 2022-07-18T03:04:11.487 INFO:teuthology.orchestra.run.smithi133.stdout: "hostnames_s3website": [], 2022-07-18T03:04:11.487 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:11.487 INFO:teuthology.orchestra.run.smithi133.stdout: "zones": [ 2022-07-18T03:04:11.487 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.487 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:11.488 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone1", 2022-07-18T03:04:11.488 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:04:11.488 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:04:11.494 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.495 INFO:teuthology.orchestra.run.smithi133.stdout: "log_meta": false, 2022-07-18T03:04:11.495 INFO:teuthology.orchestra.run.smithi133.stdout: "log_data": true, 2022-07-18T03:04:11.495 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:11.495 INFO:teuthology.orchestra.run.smithi133.stdout: "read_only": false, 2022-07-18T03:04:11.496 INFO:teuthology.orchestra.run.smithi133.stdout: "tier_type": "", 2022-07-18T03:04:11.496 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from_all": true, 2022-07-18T03:04:11.496 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from": [], 2022-07-18T03:04:11.496 INFO:teuthology.orchestra.run.smithi133.stdout: "redirect_zone": "", 2022-07-18T03:04:11.496 INFO:teuthology.orchestra.run.smithi133.stdout: "supported_features": [ 2022-07-18T03:04:11.497 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:04:11.497 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:04:11.497 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.497 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.501 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:04:11.501 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone2", 2022-07-18T03:04:11.502 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:04:11.502 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi159.front.sepia.ceph.com:8000" 2022-07-18T03:04:11.502 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.503 INFO:teuthology.orchestra.run.smithi133.stdout: "log_meta": false, 2022-07-18T03:04:11.503 INFO:teuthology.orchestra.run.smithi133.stdout: "log_data": true, 2022-07-18T03:04:11.505 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:11.505 INFO:teuthology.orchestra.run.smithi133.stdout: "read_only": false, 2022-07-18T03:04:11.505 INFO:teuthology.orchestra.run.smithi133.stdout: "tier_type": "", 2022-07-18T03:04:11.506 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from_all": true, 2022-07-18T03:04:11.506 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from": [], 2022-07-18T03:04:11.506 INFO:teuthology.orchestra.run.smithi133.stdout: "redirect_zone": "", 2022-07-18T03:04:11.506 INFO:teuthology.orchestra.run.smithi133.stdout: "supported_features": [ 2022-07-18T03:04:11.506 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:04:11.507 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:04:11.507 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.507 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.507 INFO:teuthology.orchestra.run.smithi133.stdout: "id": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T03:04:11.534 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "test-zone3", 2022-07-18T03:04:11.534 INFO:teuthology.orchestra.run.smithi133.stdout: "endpoints": [ 2022-07-18T03:04:11.534 INFO:teuthology.orchestra.run.smithi133.stdout: "http://smithi133.front.sepia.ceph.com:8001" 2022-07-18T03:04:11.538 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.538 INFO:teuthology.orchestra.run.smithi133.stdout: "log_meta": false, 2022-07-18T03:04:11.538 INFO:teuthology.orchestra.run.smithi133.stdout: "log_data": true, 2022-07-18T03:04:11.539 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:11.539 INFO:teuthology.orchestra.run.smithi133.stdout: "read_only": false, 2022-07-18T03:04:11.539 INFO:teuthology.orchestra.run.smithi133.stdout: "tier_type": "", 2022-07-18T03:04:11.539 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from_all": true, 2022-07-18T03:04:11.539 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_from": [], 2022-07-18T03:04:11.540 INFO:teuthology.orchestra.run.smithi133.stdout: "redirect_zone": "", 2022-07-18T03:04:11.540 INFO:teuthology.orchestra.run.smithi133.stdout: "supported_features": [ 2022-07-18T03:04:11.540 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:04:11.540 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:04:11.541 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:11.541 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.541 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_targets": [ 2022-07-18T03:04:11.541 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.541 INFO:teuthology.orchestra.run.smithi133.stdout: "name": "default-placement", 2022-07-18T03:04:11.542 INFO:teuthology.orchestra.run.smithi133.stdout: "tags": [], 2022-07-18T03:04:11.542 INFO:teuthology.orchestra.run.smithi133.stdout: "storage_classes": [ 2022-07-18T03:04:11.542 INFO:teuthology.orchestra.run.smithi133.stdout: "STANDARD" 2022-07-18T03:04:11.542 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:04:11.543 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:11.543 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.543 INFO:teuthology.orchestra.run.smithi133.stdout: "default_placement": "default-placement", 2022-07-18T03:04:11.543 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:11.543 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_policy": { 2022-07-18T03:04:11.563 INFO:teuthology.orchestra.run.smithi133.stdout: "groups": [] 2022-07-18T03:04:11.563 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.564 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled_features": [ 2022-07-18T03:04:11.564 INFO:teuthology.orchestra.run.smithi133.stdout: "resharding" 2022-07-18T03:04:11.565 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:04:11.565 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:11.566 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:11.566 INFO:teuthology.orchestra.run.smithi133.stdout: "short_zone_ids": [ 2022-07-18T03:04:11.567 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.567 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:11.567 INFO:teuthology.orchestra.run.smithi133.stdout: "val": 3166817753 2022-07-18T03:04:11.568 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.568 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.569 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:04:11.569 INFO:teuthology.orchestra.run.smithi133.stdout: "val": 1682003393 2022-07-18T03:04:11.570 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.570 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:11.570 INFO:teuthology.orchestra.run.smithi133.stdout: "key": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T03:04:11.571 INFO:teuthology.orchestra.run.smithi133.stdout: "val": 2112101669 2022-07-18T03:04:11.571 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:11.572 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:04:11.572 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.648 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zonegroup": "test-zonegroup", 2022-07-18T03:04:11.649 INFO:teuthology.orchestra.run.smithi133.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:11.650 INFO:teuthology.orchestra.run.smithi133.stdout: "period_config": { 2022-07-18T03:04:11.650 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_quota": { 2022-07-18T03:04:11.651 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:04:11.651 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:04:11.651 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:04:11.652 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:04:11.652 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:04:11.661 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.666 INFO:teuthology.orchestra.run.smithi133.stdout: "user_quota": { 2022-07-18T03:04:11.666 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:04:11.667 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:04:11.685 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:04:11.686 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:04:11.686 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:04:11.687 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.688 INFO:teuthology.orchestra.run.smithi133.stdout: "user_ratelimit": { 2022-07-18T03:04:11.692 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_ops": 0, 2022-07-18T03:04:11.692 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_ops": 0, 2022-07-18T03:04:11.693 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_bytes": 0, 2022-07-18T03:04:11.693 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_bytes": 0, 2022-07-18T03:04:11.694 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false 2022-07-18T03:04:11.695 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.695 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_ratelimit": { 2022-07-18T03:04:11.696 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_ops": 0, 2022-07-18T03:04:11.696 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_ops": 0, 2022-07-18T03:04:11.713 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_bytes": 0, 2022-07-18T03:04:11.714 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_bytes": 0, 2022-07-18T03:04:11.715 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false 2022-07-18T03:04:11.715 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.715 INFO:teuthology.orchestra.run.smithi133.stdout: "anonymous_ratelimit": { 2022-07-18T03:04:11.716 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_ops": 0, 2022-07-18T03:04:11.716 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_ops": 0, 2022-07-18T03:04:11.717 INFO:teuthology.orchestra.run.smithi133.stdout: "max_read_bytes": 0, 2022-07-18T03:04:11.761 INFO:teuthology.orchestra.run.smithi133.stdout: "max_write_bytes": 0, 2022-07-18T03:04:11.762 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false 2022-07-18T03:04:11.762 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:11.763 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:11.763 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:11.816 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_name": "test-realm", 2022-07-18T03:04:11.863 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:04:11.864 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:04:11.883 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-07-18T03:04:11.884 INFO:teuthology.orchestra.run:waiting for 300 2022-07-18T03:04:11.885 INFO:tasks.rgw.c1.client.1.smithi133.stdout:2022-07-18T03:04:11.877+0000 7f7530d3f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 46201) UID: 0 2022-07-18T03:04:11.893 INFO:tasks.rgw.c1.client.1.smithi133.stdout:2022-07-18T03:04:11.877+0000 7f75418bd600 -1 shutting down 2022-07-18T03:04:16.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-07-18T03:04:18.004 INFO:tasks.rgw.c1.client.1:Stopped 2022-07-18T03:04:18.005 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-07-18T03:04:18.006 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-07-18T03:04:18.011 INFO:tasks.rgw.c1.client.1:Started 2022-07-18T03:04:18.012 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001 2022-07-18T03:04:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:04:18.054 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:18.055 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:18.056 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:04:19.059 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001 2022-07-18T03:04:19.084 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:19.085 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:19.087 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-07-18T03:04:19.089 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2022-07-18T03:04:19.090 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-07-18T03:04:19.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi133.front.sepia.ceph.com:8000', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--url', 'http://smithi133.front.sepia.ceph.com:8000', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:19.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi133.front.sepia.ceph.com:8000', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--url', 'http://smithi133.front.sepia.ceph.com:8000', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:19.093 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi133.front.sepia.ceph.com:8000 --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --url http://smithi133.front.sepia.ceph.com:8000 --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:19.218 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:19.218 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:19.316 INFO:tasks.rgw.c2.client.0.smithi159.stdout:2022-07-18T03:04:19.198+0000 7ff603aac700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.66d17ce1-c2e4-4210-a2b7-a5e3888bc178.control err=-107 2022-07-18T03:04:19.318 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:19.320 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:19.321 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-realm", 2022-07-18T03:04:19.321 INFO:teuthology.orchestra.run.smithi159.stdout: "current_period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:19.321 INFO:teuthology.orchestra.run.smithi159.stdout: "epoch": 2 2022-07-18T03:04:19.321 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:19.327 INFO:tasks.rgw_multisite:creating zone.. 2022-07-18T03:04:19.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi159.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:19.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6VJB8SA94IKXZOZ9', '--secret', 'HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg', '--endpoints', 'http://smithi159.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:19.333 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 6VJB8SA94IKXZOZ9 --secret HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg --endpoints http://smithi159.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:19.400 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:19.401 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:19.452 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:04:19.419+0000 7f378e4dc540 0 failed reading obj info from .rgw.root:zone_info.9b07f48a-ef73-4ad5-94be-e184c9a423d4: (2) No such file or directory 2022-07-18T03:04:19.453 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:04:19.419+0000 7f378e4dc540 0 WARNING: could not read zone params for zone id=9b07f48a-ef73-4ad5-94be-e184c9a423d4 name=test-zone1 2022-07-18T03:04:19.453 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:04:19.420+0000 7f378e4dc540 0 failed reading obj info from .rgw.root:zone_info.ea63e6cf-483a-453e-81a7-bf1983b4a211: (2) No such file or directory 2022-07-18T03:04:19.453 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:04:19.420+0000 7f378e4dc540 0 WARNING: could not read zone params for zone id=ea63e6cf-483a-453e-81a7-bf1983b4a211 name=test-zone3 2022-07-18T03:04:19.477 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:19.524 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "a24354e3-daf6-4450-84a1-d728d5c57994", 2022-07-18T03:04:19.525 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone4", 2022-07-18T03:04:19.525 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-07-18T03:04:19.525 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "test-zone4.rgw.control", 2022-07-18T03:04:19.526 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-07-18T03:04:19.548 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-07-18T03:04:19.549 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "test-zone4.rgw.log", 2022-07-18T03:04:19.549 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-07-18T03:04:19.550 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-07-18T03:04:19.550 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-07-18T03:04:19.551 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-07-18T03:04:19.551 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-07-18T03:04:19.552 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-07-18T03:04:19.552 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-07-18T03:04:19.552 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-07-18T03:04:19.553 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-07-18T03:04:19.553 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T03:04:19.613 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:04:19.613 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:04:19.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:19.648 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T03:04:19.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:19.658 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T03:04:19.659 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:19.697 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-07-18T03:04:19.697 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T03:04:19.701 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T03:04:19.704 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2022-07-18T03:04:19.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:19.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:19.709 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-07-18T03:04:19.709 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T03:04:19.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:19.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:19.720 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:19.720 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_config": { 2022-07-18T03:04:19.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event_retention_days": 7, 2022-07-18T03:04:19.721 INFO:teuthology.orchestra.run.smithi159.stdout: "start_with_full_sync": false 2022-07-18T03:04:19.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:19.721 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:19.722 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-07-18T03:04:19.722 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:19.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2022-07-18T03:04:20.906 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'test-zone4.rgw.buckets.data' created 2022-07-18T03:04:20.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2022-07-18T03:04:21.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-07-18T03:04:21.961 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2022-07-18T03:04:21.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:21.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:21.977 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:22.036 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:22.036 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:22.072 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:22.073 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "a24354e3-daf6-4450-84a1-d728d5c57994", 2022-07-18T03:04:22.073 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone4", 2022-07-18T03:04:22.078 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-07-18T03:04:22.079 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "test-zone4.rgw.control", 2022-07-18T03:04:22.080 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-07-18T03:04:22.080 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-07-18T03:04:22.080 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "test-zone4.rgw.log", 2022-07-18T03:04:22.081 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-07-18T03:04:22.081 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-07-18T03:04:22.131 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-07-18T03:04:22.147 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-07-18T03:04:22.148 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-07-18T03:04:22.148 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-07-18T03:04:22.149 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-07-18T03:04:22.232 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-07-18T03:04:22.279 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-07-18T03:04:22.279 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T03:04:22.280 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T03:04:22.280 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T03:04:22.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:22.280 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T03:04:22.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:22.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T03:04:22.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:22.281 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-07-18T03:04:22.281 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T03:04:22.298 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T03:04:22.299 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2022-07-18T03:04:22.299 INFO:teuthology.orchestra.run.smithi159.stdout: "compression_type": "random" 2022-07-18T03:04:22.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:22.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:22.301 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-07-18T03:04:22.301 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T03:04:22.311 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:22.312 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:22.319 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:22.319 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_config": { 2022-07-18T03:04:22.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event_retention_days": 7, 2022-07-18T03:04:22.320 INFO:teuthology.orchestra.run.smithi159.stdout: "start_with_full_sync": false 2022-07-18T03:04:22.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:22.321 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "", 2022-07-18T03:04:22.331 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-07-18T03:04:22.331 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:22.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:22.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:22.334 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:22.390 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:22.391 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:22.392 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T03:04:22.391+0000 7f2c9e034540 0 period (c2173771-3b17-40d3-8c8b-0df64165bc30 does not have zone a24354e3-daf6-4450-84a1-d728d5c57994 configured 2022-07-18T03:04:28.963 INFO:teuthology.orchestra.run.smithi159.stderr:Sending period to new master zone 9b07f48a-ef73-4ad5-94be-e184c9a423d4 2022-07-18T03:04:28.991 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:28.991 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:28.992 INFO:teuthology.orchestra.run.smithi159.stdout: "epoch": 4, 2022-07-18T03:04:28.992 INFO:teuthology.orchestra.run.smithi159.stdout: "predecessor_uuid": "af1b1dd9-a63b-4529-89f0-3f46b7648654", 2022-07-18T03:04:28.993 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": [], 2022-07-18T03:04:28.993 INFO:teuthology.orchestra.run.smithi159.stdout: "period_map": { 2022-07-18T03:04:28.994 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:28.994 INFO:teuthology.orchestra.run.smithi159.stdout: "zonegroups": [ 2022-07-18T03:04:28.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:28.995 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "test-zonegroup", 2022-07-18T03:04:28.995 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zonegroup", 2022-07-18T03:04:28.996 INFO:teuthology.orchestra.run.smithi159.stdout: "api_name": "", 2022-07-18T03:04:28.996 INFO:teuthology.orchestra.run.smithi159.stdout: "is_master": true, 2022-07-18T03:04:29.010 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:04:29.011 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:04:29.011 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.012 INFO:teuthology.orchestra.run.smithi159.stdout: "hostnames": [], 2022-07-18T03:04:29.012 INFO:teuthology.orchestra.run.smithi159.stdout: "hostnames_s3website": [], 2022-07-18T03:04:29.012 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:29.013 INFO:teuthology.orchestra.run.smithi159.stdout: "zones": [ 2022-07-18T03:04:29.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.017 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:29.017 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone1", 2022-07-18T03:04:29.018 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:04:29.018 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T03:04:29.019 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.019 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T03:04:29.020 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T03:04:29.020 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:29.020 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T03:04:29.021 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T03:04:29.021 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T03:04:29.021 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T03:04:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T03:04:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T03:04:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:04:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "a24354e3-daf6-4450-84a1-d728d5c57994", 2022-07-18T03:04:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone4", 2022-07-18T03:04:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:04:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi159.front.sepia.ceph.com:8001" 2022-07-18T03:04:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T03:04:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T03:04:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T03:04:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "pubsub", 2022-07-18T03:04:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": false, 2022-07-18T03:04:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [ 2022-07-18T03:04:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "test-zone1" 2022-07-18T03:04:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T03:04:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T03:04:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:04:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:04:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone2", 2022-07-18T03:04:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:04:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi159.front.sepia.ceph.com:8000" 2022-07-18T03:04:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T03:04:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T03:04:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T03:04:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T03:04:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T03:04:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T03:04:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T03:04:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T03:04:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:04:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T03:04:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone3", 2022-07-18T03:04:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T03:04:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8001" 2022-07-18T03:04:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T03:04:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T03:04:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T03:04:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T03:04:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T03:04:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T03:04:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T03:04:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T03:04:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T03:04:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:04:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_targets": [ 2022-07-18T03:04:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "default-placement", 2022-07-18T03:04:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "tags": [], 2022-07-18T03:04:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": [ 2022-07-18T03:04:29.131 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD" 2022-07-18T03:04:29.131 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.132 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:29.132 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.133 INFO:teuthology.orchestra.run.smithi159.stdout: "default_placement": "default-placement", 2022-07-18T03:04:29.145 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:29.147 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_policy": { 2022-07-18T03:04:29.149 INFO:teuthology.orchestra.run.smithi159.stdout: "groups": [] 2022-07-18T03:04:29.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.149 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled_features": [ 2022-07-18T03:04:29.150 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T03:04:29.150 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.151 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:29.151 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T03:04:29.152 INFO:teuthology.orchestra.run.smithi159.stdout: "short_zone_ids": [ 2022-07-18T03:04:29.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.152 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:29.153 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 3166817753 2022-07-18T03:04:29.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.166 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "a24354e3-daf6-4450-84a1-d728d5c57994", 2022-07-18T03:04:29.166 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 2779073914 2022-07-18T03:04:29.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.168 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T03:04:29.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 1682003393 2022-07-18T03:04:29.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:29.172 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T03:04:29.172 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 2112101669 2022-07-18T03:04:29.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:29.173 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:29.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.174 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zonegroup": "test-zonegroup", 2022-07-18T03:04:29.175 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T03:04:29.176 INFO:teuthology.orchestra.run.smithi159.stdout: "period_config": { 2022-07-18T03:04:29.176 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_quota": { 2022-07-18T03:04:29.177 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false, 2022-07-18T03:04:29.177 INFO:teuthology.orchestra.run.smithi159.stdout: "check_on_raw": false, 2022-07-18T03:04:29.178 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size": -1, 2022-07-18T03:04:29.183 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size_kb": 0, 2022-07-18T03:04:29.184 INFO:teuthology.orchestra.run.smithi159.stdout: "max_objects": -1 2022-07-18T03:04:29.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.186 INFO:teuthology.orchestra.run.smithi159.stdout: "user_quota": { 2022-07-18T03:04:29.186 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false, 2022-07-18T03:04:29.187 INFO:teuthology.orchestra.run.smithi159.stdout: "check_on_raw": false, 2022-07-18T03:04:29.187 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size": -1, 2022-07-18T03:04:29.187 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size_kb": 0, 2022-07-18T03:04:29.188 INFO:teuthology.orchestra.run.smithi159.stdout: "max_objects": -1 2022-07-18T03:04:29.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.189 INFO:teuthology.orchestra.run.smithi159.stdout: "user_ratelimit": { 2022-07-18T03:04:29.189 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T03:04:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T03:04:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T03:04:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T03:04:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T03:04:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.192 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_ratelimit": { 2022-07-18T03:04:29.192 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T03:04:29.193 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T03:04:29.193 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T03:04:29.193 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T03:04:29.194 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T03:04:29.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.195 INFO:teuthology.orchestra.run.smithi159.stdout: "anonymous_ratelimit": { 2022-07-18T03:04:29.195 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T03:04:29.196 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T03:04:29.196 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T03:04:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T03:04:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T03:04:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:29.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:29.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T03:04:29.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_name": "test-realm", 2022-07-18T03:04:29.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:29.200 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:29.201 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2022-07-18T03:04:29.202 INFO:teuthology.orchestra.run:waiting for 300 2022-07-18T03:04:29.204 INFO:tasks.rgw.c2.client.1.smithi159.stdout:2022-07-18T03:04:29.201+0000 7fe4de060700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 46061) UID: 0 2022-07-18T03:04:29.204 INFO:tasks.rgw.c2.client.1.smithi159.stdout:2022-07-18T03:04:29.201+0000 7fe4eebde600 -1 shutting down 2022-07-18T03:04:31.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2022-07-18T03:04:35.335 INFO:tasks.rgw.c2.client.1:Stopped 2022-07-18T03:04:35.336 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-07-18T03:04:35.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-07-18T03:04:35.348 INFO:tasks.rgw.c2.client.1:Started 2022-07-18T03:04:35.349 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8001 2022-07-18T03:04:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:04:35.417 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:35.417 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:35.417 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi159.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:04:36.490 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8001 2022-07-18T03:04:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:04:36.539 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:36.540 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:36.540 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi159.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:04:37.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2022-07-18T03:04:38.541 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8001 2022-07-18T03:04:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-07-18T03:04:38.580 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:38.586 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:38.586 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi159.front.sepia.ceph.com port 8001: Connection refused 2022-07-18T03:04:42.600 DEBUG:teuthology.orchestra.run.smithi159:> curl http://smithi159.front.sepia.ceph.com:8001 2022-07-18T03:04:42.686 INFO:teuthology.orchestra.run.smithi159.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-07-18T03:04:42.687 INFO:teuthology.orchestra.run.smithi159.stderr: Dload Upload Total Spent Left Speed 2022-07-18T03:04:42.687 INFO:teuthology.orchestra.run.smithi159.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2022-07-18T03:04:42.690 INFO:teuthology.orchestra.run.smithi159.stdout:{"Code":"MethodNotAllowed","RequestId":"tx00000543dfe46130aaf3d-0062d4cdca-10e8-test-zone4","HostId":"10e8-test-zone4-test-zonegroup"} 2022-07-18T03:04:42.700 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2022-07-18T03:04:42.700 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2022-07-18T03:04:42.832 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa/../src/test/rgw/rgw_multi 2022-07-18T03:04:42.949 INFO:tasks.rgw_multisite_tests:creating test user.. 2022-07-18T03:04:42.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:42.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T03:04:42.952 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T03:04:43.019 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:43.019 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:43.080 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:04:43.080 INFO:teuthology.orchestra.run.smithi133.stdout: "user_id": "rgw-multisite-test-user", 2022-07-18T03:04:43.081 INFO:teuthology.orchestra.run.smithi133.stdout: "display_name": "Multisite Test User", 2022-07-18T03:04:43.081 INFO:teuthology.orchestra.run.smithi133.stdout: "email": "", 2022-07-18T03:04:43.086 INFO:teuthology.orchestra.run.smithi133.stdout: "suspended": 0, 2022-07-18T03:04:43.087 INFO:teuthology.orchestra.run.smithi133.stdout: "max_buckets": 1000, 2022-07-18T03:04:43.087 INFO:teuthology.orchestra.run.smithi133.stdout: "subusers": [], 2022-07-18T03:04:43.088 INFO:teuthology.orchestra.run.smithi133.stdout: "keys": [ 2022-07-18T03:04:43.089 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:43.089 INFO:teuthology.orchestra.run.smithi133.stdout: "user": "rgw-multisite-test-user", 2022-07-18T03:04:43.090 INFO:teuthology.orchestra.run.smithi133.stdout: "access_key": "1ZE2DVR9X6CB66IYMJG4", 2022-07-18T03:04:43.090 INFO:teuthology.orchestra.run.smithi133.stdout: "secret_key": "ygnvowyWVbWXRy3jjt5P9F8Cl1YXRUSUcSlXKIQb" 2022-07-18T03:04:43.091 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:43.091 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:43.092 INFO:teuthology.orchestra.run.smithi133.stdout: "swift_keys": [], 2022-07-18T03:04:43.092 INFO:teuthology.orchestra.run.smithi133.stdout: "caps": [ 2022-07-18T03:04:43.093 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:43.093 INFO:teuthology.orchestra.run.smithi133.stdout: "type": "roles", 2022-07-18T03:04:43.094 INFO:teuthology.orchestra.run.smithi133.stdout: "perm": "*" 2022-07-18T03:04:43.094 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:43.100 INFO:teuthology.orchestra.run.smithi133.stdout: ], 2022-07-18T03:04:43.100 INFO:teuthology.orchestra.run.smithi133.stdout: "op_mask": "read, write, delete", 2022-07-18T03:04:43.101 INFO:teuthology.orchestra.run.smithi133.stdout: "default_placement": "", 2022-07-18T03:04:43.102 INFO:teuthology.orchestra.run.smithi133.stdout: "default_storage_class": "", 2022-07-18T03:04:43.102 INFO:teuthology.orchestra.run.smithi133.stdout: "placement_tags": [], 2022-07-18T03:04:43.102 INFO:teuthology.orchestra.run.smithi133.stdout: "bucket_quota": { 2022-07-18T03:04:43.103 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:04:43.103 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:04:43.103 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:04:43.103 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:04:43.104 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:04:43.104 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:43.104 INFO:teuthology.orchestra.run.smithi133.stdout: "user_quota": { 2022-07-18T03:04:43.104 INFO:teuthology.orchestra.run.smithi133.stdout: "enabled": false, 2022-07-18T03:04:43.105 INFO:teuthology.orchestra.run.smithi133.stdout: "check_on_raw": false, 2022-07-18T03:04:43.105 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size": -1, 2022-07-18T03:04:43.105 INFO:teuthology.orchestra.run.smithi133.stdout: "max_size_kb": 0, 2022-07-18T03:04:43.105 INFO:teuthology.orchestra.run.smithi133.stdout: "max_objects": -1 2022-07-18T03:04:43.105 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:43.106 INFO:teuthology.orchestra.run.smithi133.stdout: "temp_url_keys": [], 2022-07-18T03:04:43.106 INFO:teuthology.orchestra.run.smithi133.stdout: "type": "rgw", 2022-07-18T03:04:43.106 INFO:teuthology.orchestra.run.smithi133.stdout: "mfa_ids": [] 2022-07-18T03:04:43.106 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:04:43.107 INFO:teuthology.orchestra.run.smithi133.stdout: 2022-07-18T03:04:43.109 INFO:rgw_multi.tests:meta checkpoint 2022-07-18T03:04:43.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:43.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:43.111 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:04:43.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:43.162 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:43.221 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:04:43.243 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:04:43.244 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:43.244 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:04:43.244 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:04:43.245 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:43.245 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:43.245 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:04:43.246 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:04:43.254 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:43.254 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:43.255 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:04:43.255 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:04:43.255 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:04:43.255 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:04:43.256 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:04:43.256 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:04:43.256 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:04:43.257 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:04:43.257 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:04:43.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:43.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:43.268 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:04:43.330 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:43.340 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:43.403 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:43.404 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:04:43.404 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:04:43.405 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:04:43.405 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:04:43.405 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:43.407 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:43.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:43.407 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:04:43.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:43.408 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:04:43.409 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:43.422 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:43.422 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:43.423 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:43.423 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:04:43.424 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:43.424 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:43.425 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:43.425 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:43.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:43.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:43.443 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:04:43.444 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:43.444 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:43.445 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:43.445 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:43.445 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:43.510 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:43.511 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:43.511 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:43.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:43.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:43.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:43.513 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:04:43.513 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:43.513 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:43.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:43.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:43.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:43.534 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:43.535 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:43.535 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:43.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:43.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:43.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:43.537 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:04:43.537 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:43.538 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:43.538 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:43.539 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:43.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:43.539 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:43.540 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:43.540 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:43.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:43.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:43.541 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:04:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:04:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:04:43.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:43.565 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:04:43Z" 2022-07-18T03:04:43.571 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:43.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:04:48.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:48.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:48.578 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:04:48.638 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:48.639 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:48.691 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:04:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:04:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:04:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:04:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:04:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:04:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:04:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:04:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:04:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:04:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:04:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:04:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:04:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:04:48Z" 2022-07-18T03:04:48.742 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:48.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:04:53.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:53.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:53.746 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:04:53.802 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:53.803 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:53.882 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:04:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:04:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:04:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:04:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:04:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:04:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:04:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:53.887 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:53.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:04:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:53.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:53.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:04:53.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:53.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:53.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:53.894 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:53.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:53.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:04:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:53.897 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:53.900 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:53.900 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:53.901 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:53.901 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:53.901 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:53.902 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:53.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:53.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:53.902 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:53.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:53.907 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:04:53.907 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:04:53.908 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:04:53.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:53.909 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:04:53Z" 2022-07-18T03:04:53.910 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:53.913 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:04:58.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:58.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:04:58.916 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:04:58.973 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:04:58.973 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:04:59.047 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:04:59.048 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:04:59.048 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:04:59.049 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:04:59.049 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:04:59.049 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:04:59.050 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:59.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:59.051 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:04:59.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:59.053 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:04:59.054 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:59.054 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:59.055 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:59.055 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:59.055 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:04:59.055 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:59.056 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:59.056 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:59.056 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:59.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:59.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:59.060 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:04:59.060 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:59.060 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:59.061 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:59.061 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:59.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:59.063 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:59.064 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:59.064 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:59.065 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:59.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:59.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:59.066 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:04:59.066 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:59.067 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:59.068 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:59.069 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:59.069 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:59.069 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:59.069 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:59.069 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:59.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:59.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:59.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:04:59.072 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:04:59.073 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:04:59.073 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:04:59.074 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:04:59.074 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:04:59.074 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:04:59.074 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:04:59.075 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:04:59.075 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:04:59.075 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:59.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:04:59.076 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:04:59.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:59.077 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:04:59.077 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:04:59.077 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:04:59.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:04:59.078 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:04:59Z" 2022-07-18T03:04:59.078 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:04:59.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:04.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:04.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:04.084 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:04.140 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:04.141 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:04.197 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:04.198 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:04.199 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:04.199 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:04.200 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:04.200 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:04.201 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:04.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:04.202 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:04.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:04.202 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:04.203 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:04.203 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:04.204 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:04.204 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:04.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:04.205 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:04.206 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:04.207 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:04.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:04.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:04.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:04.209 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:04.209 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:04.210 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:04.210 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:04.211 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:04.211 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:04.213 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:04.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:04.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:04.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:04.215 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:04.217 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:04.217 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:04.217 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:04.217 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:04.218 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:04.218 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:04.218 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:04.218 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:04.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:04.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:04.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:04.220 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:04.220 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:04.220 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:04.220 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:04.221 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:04.221 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:04.221 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:04.221 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:04.222 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:04.222 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:04.222 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:04.222 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:04.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:04.223 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:04.223 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:04.223 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:04.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:04.224 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:04Z" 2022-07-18T03:05:04.224 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:04.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:09.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:09.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:09.229 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:09.284 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:09.284 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:09.355 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:09Z" 2022-07-18T03:05:09.371 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:09.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:14.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:14.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:14.373 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:14.429 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:14.430 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:14.509 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:14.509 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:14.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:14.510 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:14.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:14.511 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:14.514 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:14.514 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:14.514 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:14.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:14.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:14.516 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:14.517 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:14.517 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:14.518 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:14.518 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:14.519 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:14.519 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:14.520 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:14.520 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:14.520 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:14.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:14.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:14.523 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:14.524 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:14.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:14.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:14.526 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:14.526 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:14.527 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:14.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:14.528 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:14.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:14.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:14.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:14.529 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:14.530 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:14.530 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:14.531 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:14.531 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:14.532 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:14.533 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:14.533 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:14.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:14.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:14.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:14.535 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:14.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:14.536 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:14.537 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:14.537 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:14.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:14.538 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:14Z" 2022-07-18T03:05:14.538 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:14.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:19.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:19.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:19.544 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:19.609 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:19.610 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:19.678 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:19.679 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:19.679 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:19.679 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:19.682 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:19.682 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:19.684 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:19.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:19.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:19.685 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:19.685 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:19.685 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:19.685 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:19.686 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:19.686 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:19.686 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:19.687 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:19.687 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:19.687 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:19.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:19.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:19.688 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:19.688 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:19.688 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:19.689 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:19.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:19.689 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:19.689 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:19.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:19.690 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:19.690 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:19.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:19.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:19.691 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:19.691 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:19.691 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:19.692 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:19.692 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:19.692 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:19.692 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:19.693 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:19.693 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:19.693 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:19.693 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:19.694 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:19.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:19.694 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:19.694 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:19.695 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:19.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:19.696 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:19Z" 2022-07-18T03:05:19.696 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:19.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:24.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:24.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:24.702 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:24.759 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:24.759 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:24.813 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:24Z" 2022-07-18T03:05:24.838 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:24.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:29.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:29.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:29.843 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:29.898 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:29.898 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:29.967 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:29.968 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:29.968 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:29.968 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:29.969 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:29.969 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:29.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:29.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:29.971 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:29.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:29.971 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:29.972 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:29.972 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:29.973 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:29.974 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:29.974 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:29.974 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:29.975 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:29.975 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:29.976 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:29.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:29.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:29.978 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:29.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:29.979 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:29.980 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:29.980 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:29.981 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:29.990 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:29.990 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:29Z" 2022-07-18T03:05:29.991 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:29.993 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:34.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:34.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:34.995 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:35.051 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:35.052 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:35.114 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:35.115 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:35.115 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:35.116 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:35.116 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:35.117 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:35.117 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:35.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:35.118 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:35.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:35.119 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:35.120 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:35.120 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:35.121 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:35.121 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:35.122 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:35.122 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:35.123 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:35.123 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:35.123 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:35.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:35.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:35.125 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:35.126 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:35.126 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:35.126 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:35.127 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:35.127 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:35.128 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:35.128 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:35.128 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:35.129 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:35.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:35.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:35.130 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:35.131 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:35.131 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:35.131 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:35.132 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:35.132 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:35.132 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:35.133 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:35.133 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:35.134 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:35.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:35.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:35.135 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:35.135 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:35.136 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:35.136 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:35.137 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:35.137 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:35.140 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:35.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:35.141 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:35.141 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:35.141 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:35.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:35.142 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:35Z" 2022-07-18T03:05:35.143 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:35.145 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:40.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:40.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:40.148 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:40.205 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:40.205 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:40.278 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:40.279 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:40.279 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:40.280 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:40.280 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:40.280 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:40.281 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:40.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:40.282 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:40.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:40.282 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:40.283 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:40.283 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:40.284 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:40.284 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:40.285 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:40.285 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:40.285 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:40.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:40.286 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:40.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:40.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:40.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:40.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:40.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:40.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:40.288 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:40.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:40.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:40.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:40.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:40.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:40.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:40.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:40.292 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:40.292 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:40.292 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:40.292 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:40.293 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:40.293 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:40.293 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:40.293 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:40.294 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:40.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:40.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:40.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:40.295 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:40.295 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:40.295 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:40.295 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:40.295 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:40.296 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:40.296 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:40.296 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:40.299 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:40Z" 2022-07-18T03:05:40.299 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:40.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:45.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:45.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:45.303 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:45.359 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:45.359 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:45.442 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:45.443 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:45.443 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:45.443 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:45.444 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:45.444 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:45.444 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:45.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:45.445 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:45.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:45.446 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:45.446 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:45.446 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:45.447 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:45.447 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:45.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:45.448 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:45.448 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:45.448 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:45.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:45.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:45.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:45.449 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:45.449 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:45.450 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:45.450 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:45.450 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:45.450 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:45.451 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:45.451 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:45.452 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:45.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:45.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:45.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:45.456 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:45.456 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:45.456 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:45.457 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:45.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:45.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:45.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:45.458 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:45.458 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:45.459 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:45.459 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:45.462 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:45.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:45.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:45.464 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:45Z" 2022-07-18T03:05:45.467 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:45.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:50.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:50.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:50.473 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:50.530 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:50.531 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:50.601 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:50.601 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:50.602 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:50.602 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:50.603 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:50.603 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:50.603 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:50.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:50.604 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:50.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:50.605 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:50.606 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:50.606 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:50.607 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:50.607 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:50.607 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:50.608 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:50.608 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:50.609 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:50.609 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:50.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:50.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:50.610 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:50.613 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:50Z" 2022-07-18T03:05:50.625 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:50.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:05:55.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:55.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:05:55.631 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:05:55.688 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:05:55.688 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:05:55.772 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:05:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:05:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:05:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:05:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:05:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:05:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:05:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:05:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:05:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:05:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:55.783 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:55.783 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:55.783 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:55.783 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:55.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:55.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:55.784 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:05:55.784 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:55.785 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:55.785 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:55.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:55.785 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:55.785 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:55.786 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:55.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:55.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:55.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:05:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:05:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:05:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:05:55.788 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:05:55.788 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:05:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:05:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:05:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:05:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:05:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:05:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:05:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:05:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:05:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:05:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:05:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:05:55Z" 2022-07-18T03:05:55.792 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:05:55.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:00.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:00.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:00.800 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:00.856 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:00.856 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:00.924 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:00.925 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:00.925 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:00.925 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:00.925 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:00.926 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:00.926 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:00.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:00.926 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:00.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:00.927 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:00.927 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:00.927 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:00.928 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:00.928 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:00.928 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:00.928 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:00.929 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:00.929 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:00.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:00.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:00.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:00.931 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:00.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:00.932 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:00.932 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:00.933 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:00.933 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:00.933 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:00.934 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:00.934 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:00.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:00.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:00.937 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:00.937 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:00.938 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:00.938 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:00.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:00.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:00.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:00.941 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:00.941 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:00.942 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:00.942 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:00.944 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:00.944 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:00.945 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:00.945 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:00.945 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:00.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:00.946 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:00.947 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:00.947 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:00.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:00.948 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:00Z" 2022-07-18T03:06:00.948 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:00.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:05.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:05.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:05.955 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:06.013 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:06.014 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:06.100 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:06.100 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:06.101 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:06.102 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:06.102 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:06.103 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:06.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:06.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:06.105 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:06.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:06.106 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:06.107 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:06.107 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:06.108 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:06.109 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:06.109 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:06.110 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:06.110 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:06.110 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:06.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:06.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:06.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:06.112 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:06.112 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:06.113 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:06.113 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:06.113 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:06.114 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:06.114 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:06.114 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:06.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:06.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:06.116 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:06.117 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:06.117 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:06.118 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:06.118 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:06.119 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:06.120 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:06.120 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:06.121 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:06.121 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:06.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:06.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:06.122 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:06.123 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:06.123 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:06.124 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:06.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:06.124 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:06.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:06.125 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:06.125 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:06.126 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:06.126 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:06.126 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:06.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:06.127 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:06.128 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:06.128 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:06.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:06.129 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:06Z" 2022-07-18T03:06:06.130 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:06.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:11.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:11.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:11.133 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:11.190 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:11.191 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:11.258 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:11.258 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:11.259 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:11.259 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:11.259 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:11.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:11.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:11.261 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:11.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:11.261 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:11.262 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:11.265 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:11.267 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:11.270 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:11.270 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:11.271 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:11.271 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:11.271 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:11.271 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:11.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:11.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:11.272 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:11.274 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:11.275 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:11.275 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:11.275 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:11.276 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:11.276 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:11.277 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:11.277 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:11.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:11.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:11.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:11.279 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:11.280 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:11.280 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:11.281 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:11.281 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:11.282 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:11.282 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:11.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:11.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:11.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:11.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:11.284 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:11.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:11.286 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:11.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:11.287 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:11.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:11.288 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:11Z" 2022-07-18T03:06:11.288 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:11.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:16.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:16.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:16.300 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:16.355 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:16.356 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:16.434 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:16.435 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:16.435 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:16.436 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:16.436 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:16.437 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:16.437 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:16.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:16.438 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:16.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:16.439 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:16.439 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:16.440 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:16.440 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:16.441 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:16.441 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:16.441 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:16.442 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:16.443 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:16.443 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:16.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:16.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:16.445 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:16.445 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:16.446 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:16.446 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:16.447 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:16.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:16.448 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:16.448 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:16.449 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:16.449 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:16.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:16.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:16.450 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:16.451 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:16.451 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:16.452 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:16.452 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:16.452 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:16.453 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:16.454 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:16.454 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:16.455 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:16.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:16.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:16.456 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:16.457 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:16.457 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:16.457 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:16.458 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:16.459 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:16.459 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:16.459 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:16.460 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:16.460 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:16.461 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:16.461 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:16.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:16.462 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:16.463 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:16.463 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:16.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:16Z" 2022-07-18T03:06:16.465 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:16.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:21.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:21.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:21.472 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:21.529 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:21.529 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:21.585 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:21.585 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:21.585 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:21.586 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:21.586 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:21.586 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:21.586 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:21.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:21.587 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:21.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:21.587 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:21.588 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:21.588 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:21.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:21.588 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:21.589 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:21.589 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:21.589 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:21.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:21.589 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:21.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:21.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:21.592 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:21.592 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:21.592 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:21.592 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:21.593 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:21.593 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:21.593 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:21.593 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:21.594 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:21.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:21.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:21.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:21.595 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:21.596 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:21.596 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:21.596 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:21.596 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:21.597 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:21.597 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:21.597 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:21.597 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:21.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:21.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:21.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:21.598 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:21.599 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:21.599 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:21.599 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:21.599 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:21.600 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:21.600 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:21.600 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:21.600 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:21.603 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:21.603 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:21.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:21.604 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:21Z" 2022-07-18T03:06:21.604 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:21.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:26.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:26.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:26.609 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:26.665 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:26.666 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:26.736 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:26.737 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:26.737 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:26.738 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:26.738 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:26.739 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:26.739 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:26.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:26.740 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:26.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:26.741 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:26.741 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:26.742 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:26.742 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:26.743 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:26.744 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:26.744 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:26.745 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:26.745 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:26.746 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:26.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:26.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:26.747 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:26.748 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:26.748 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:26.749 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:26.749 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:26.750 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:26.750 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:26.750 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:26.751 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:26.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:26.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:26.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:26.753 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:26.753 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:26.754 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:26.754 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:26.754 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:26.755 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:26.755 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:26.755 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:26.755 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:26.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:26.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:26.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:26.756 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:26.757 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:26.757 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:26.757 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:26.757 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:26.757 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:26.758 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:26.758 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:26.758 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:26.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:26.759 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:26.759 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:26.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:26.759 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:26.760 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:26.760 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:26.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:26.761 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:26Z" 2022-07-18T03:06:26.761 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:26.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1658113483.073958_74.1 target= 2022-07-18T03:06:31.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:31.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:31.765 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:31.824 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:31.824 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:31.884 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:31.885 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:31.885 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:31.886 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:31.886 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:31.887 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:31.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:31.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:31.888 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:31.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:31.888 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:31.889 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:31.889 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:31.889 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:31.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:31.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:31.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:31.891 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:31.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:31.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:31.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:31.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:31.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:31.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:31.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:31.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:31.894 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:31.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:31.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:31.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:31.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:31.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:31.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:31.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:31.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:31.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:31.896 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:31.896 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:31.897 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:31.897 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:31.897 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:31.897 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:31.898 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:31.898 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:31.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:31.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:31.899 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:31.899 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:31.900 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:31.900 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:31.900 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:31.900 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:31.901 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:31.901 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:31.901 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:31.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:31.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:31.902 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:31.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:31.902 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:31.903 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:31.903 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:31.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:31.903 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:31Z" 2022-07-18T03:06:31.904 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:31.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-07-18T03:06:31.906 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-07-18T03:06:31.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:31.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:31.908 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:31.974 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:31.975 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:32.047 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:06:32.047 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": { 2022-07-18T03:06:32.048 INFO:teuthology.orchestra.run.smithi133.stdout: "info": { 2022-07-18T03:06:32.048 INFO:teuthology.orchestra.run.smithi133.stdout: "status": "sync", 2022-07-18T03:06:32.049 INFO:teuthology.orchestra.run.smithi133.stdout: "num_shards": 4, 2022-07-18T03:06:32.049 INFO:teuthology.orchestra.run.smithi133.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:32.050 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.051 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.051 INFO:teuthology.orchestra.run.smithi133.stdout: "markers": [ 2022-07-18T03:06:32.051 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.051 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 0, 2022-07-18T03:06:32.052 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.052 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.052 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:32.053 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.053 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 1, 2022-07-18T03:06:32.053 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.053 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:32.054 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.054 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.054 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.055 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.055 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 1, 2022-07-18T03:06:32.056 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.056 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.056 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.057 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.057 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:32.058 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.058 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.063 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.065 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.066 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.066 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.067 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 2, 2022-07-18T03:06:32.067 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.068 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.068 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.069 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.069 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:32.070 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.070 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.071 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.072 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.072 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.073 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.073 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 3, 2022-07-18T03:06:32.074 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.074 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.076 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.076 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.077 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:32.077 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.078 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.078 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.078 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.079 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.080 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:06:32.080 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.081 INFO:teuthology.orchestra.run.smithi133.stdout: "full_sync": { 2022-07-18T03:06:32.083 INFO:teuthology.orchestra.run.smithi133.stdout: "total": 1, 2022-07-18T03:06:32.083 INFO:teuthology.orchestra.run.smithi133.stdout: "complete": 1 2022-07-18T03:06:32.083 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.083 INFO:teuthology.orchestra.run.smithi133.stdout: "current_time": "2022-07-18T03:06:32Z" 2022-07-18T03:06:32.084 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:06:32.084 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-07-18T03:06:32.085 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-07-18T03:06:32.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.086 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:32.150 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:32.151 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:32.208 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.241 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:32.242 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.246 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.246 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.246 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.252 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.253 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.253 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.254 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.255 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:32.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.257 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:32.257 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:32.257 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:32.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.258 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:32Z" 2022-07-18T03:06:32.258 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:32.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-07-18T03:06:32.261 INFO:rgw_multi.tests:meta checkpoint 2022-07-18T03:06:32.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.264 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:32.316 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:32.317 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:32.359 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:06:32.364 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:06:32.364 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.365 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:32.365 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:06:32.365 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.366 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.366 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.366 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:32.366 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.367 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.367 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.367 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:32.367 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.368 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.368 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.369 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:32.369 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.373 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:06:32.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:06:32.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.377 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:32.428 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:32.429 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:32.516 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:32.516 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:32.517 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:32.517 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:32.517 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.521 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:32.521 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.522 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:32.522 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.524 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.524 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.525 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.525 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.525 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.525 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.526 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:32.526 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.527 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.527 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.527 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.528 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.528 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.528 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.529 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:32.529 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.529 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.530 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.530 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.530 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.531 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:32.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.532 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:32.532 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:32.532 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:32.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.533 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:32Z" 2022-07-18T03:06:32.533 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:32.533 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-07-18T03:06:32.534 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-07-18T03:06:32.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.535 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:32.584 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:32.584 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:32.635 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:06:32.636 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": { 2022-07-18T03:06:32.636 INFO:teuthology.orchestra.run.smithi133.stdout: "info": { 2022-07-18T03:06:32.637 INFO:teuthology.orchestra.run.smithi133.stdout: "status": "sync", 2022-07-18T03:06:32.637 INFO:teuthology.orchestra.run.smithi133.stdout: "num_shards": 4, 2022-07-18T03:06:32.637 INFO:teuthology.orchestra.run.smithi133.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:32.638 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.638 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.638 INFO:teuthology.orchestra.run.smithi133.stdout: "markers": [ 2022-07-18T03:06:32.639 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.639 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 0, 2022-07-18T03:06:32.639 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.644 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.645 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:32.646 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.646 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 1, 2022-07-18T03:06:32.647 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.647 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:32.647 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.648 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.648 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.648 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.649 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 1, 2022-07-18T03:06:32.649 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.650 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.650 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.651 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.651 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:32.652 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.652 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.653 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.653 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.653 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.654 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.656 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 2, 2022-07-18T03:06:32.657 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.657 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.658 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.658 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.659 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:32.659 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.659 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.660 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.662 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.663 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.663 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:32.664 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 3, 2022-07-18T03:06:32.664 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:32.664 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:32.664 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:32.665 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:32.665 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:32.665 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:32.665 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.666 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:32.666 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.667 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:32.667 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:06:32.667 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.667 INFO:teuthology.orchestra.run.smithi133.stdout: "full_sync": { 2022-07-18T03:06:32.668 INFO:teuthology.orchestra.run.smithi133.stdout: "total": 1, 2022-07-18T03:06:32.668 INFO:teuthology.orchestra.run.smithi133.stdout: "complete": 1 2022-07-18T03:06:32.668 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:32.668 INFO:teuthology.orchestra.run.smithi133.stdout: "current_time": "2022-07-18T03:06:32Z" 2022-07-18T03:06:32.669 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:06:32.670 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-07-18T03:06:32.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-07-18T03:06:32.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:32.674 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:32.726 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:32.727 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:32.778 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:32.779 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:32.779 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:32.779 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:32.779 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:32.780 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:32.780 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.780 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:32.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.781 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:32.781 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.781 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.781 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:32.782 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:32.782 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.783 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:32.783 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.783 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.784 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:32.784 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.784 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.784 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.785 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.785 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.785 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.785 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.788 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:32.788 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.788 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.788 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.789 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.789 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.789 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.789 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:32.790 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:32.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:32.791 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:32.791 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:32.791 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:32.792 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:32.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:32.793 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:32.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:32.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:32.794 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:32.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.794 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:32.795 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:32.795 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:32.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:32.795 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:32Z" 2022-07-18T03:06:32.796 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:32.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-07-18T03:06:32.796 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_ff10064ab878e58afb1d2538b2086a2d4a41c017/qa/../src/test/rgw/rgw_multi' with args=['-a', '!bucket_sync_disable,!bucket_reshard', 'tests.py', 'tests_ps.py'] 2022-07-18T03:06:32.928 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.969 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:32.971 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.975 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:32.979 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.981 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:32.986 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.987 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:32.989 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.992 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:32.995 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.996 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:32.998 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:32.999 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:33.000 DEBUG:boto:Using access key provided by client. 2022-07-18T03:06:33.002 DEBUG:boto:Using secret key provided by client. 2022-07-18T03:06:33.003 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-1 2022-07-18T03:06:33.004 DEBUG:boto:path=/mlvank-1/ 2022-07-18T03:06:33.005 DEBUG:boto:auth_path=/mlvank-1/ 2022-07-18T03:06:33.006 DEBUG:boto:Method: PUT 2022-07-18T03:06:33.007 DEBUG:boto:Path: /mlvank-1/ 2022-07-18T03:06:33.008 DEBUG:boto:Data: 2022-07-18T03:06:33.009 DEBUG:boto:Headers: {} 2022-07-18T03:06:33.010 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:33.011 DEBUG:boto:Port: 8000 2022-07-18T03:06:33.012 DEBUG:boto:Params: {} 2022-07-18T03:06:33.012 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:06:33.013 DEBUG:boto:Token: None 2022-07-18T03:06:33.014 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:33 GMT /mlvank-1/ 2022-07-18T03:06:33.021 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QGMVD6uo9+ZMCH3E44YINwhB6Pk= 2022-07-18T03:06:33.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QGMVD6uo9+ZMCH3E44YINwhB6Pk=', 'Content-Length': '0'} 2022-07-18T03:06:35.425 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000840d5203fcd41426-0062d4ce39-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:35 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:35.426 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-2 2022-07-18T03:06:35.427 DEBUG:boto:path=/mlvank-2/ 2022-07-18T03:06:35.427 DEBUG:boto:auth_path=/mlvank-2/ 2022-07-18T03:06:35.428 DEBUG:boto:Method: PUT 2022-07-18T03:06:35.428 DEBUG:boto:Path: /mlvank-2/ 2022-07-18T03:06:35.429 DEBUG:boto:Data: 2022-07-18T03:06:35.429 DEBUG:boto:Headers: {} 2022-07-18T03:06:35.430 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:35.430 DEBUG:boto:Port: 8000 2022-07-18T03:06:35.431 DEBUG:boto:Params: {} 2022-07-18T03:06:35.431 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:06:35.432 DEBUG:boto:Token: None 2022-07-18T03:06:35.432 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:35 GMT /mlvank-2/ 2022-07-18T03:06:35.434 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:enppo2gxJuf4ONpf3Gr/xwLQvqo= 2022-07-18T03:06:35.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:35 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:enppo2gxJuf4ONpf3Gr/xwLQvqo=', 'Content-Length': '0'} 2022-07-18T03:06:37.984 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000074fc39fee29800f3-0062d4ce3b-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:37 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:37.985 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-3 2022-07-18T03:06:37.986 DEBUG:boto:path=/mlvank-3/ 2022-07-18T03:06:37.986 DEBUG:boto:auth_path=/mlvank-3/ 2022-07-18T03:06:37.987 DEBUG:boto:Method: PUT 2022-07-18T03:06:37.987 DEBUG:boto:Path: /mlvank-3/ 2022-07-18T03:06:37.988 DEBUG:boto:Data: 2022-07-18T03:06:37.988 DEBUG:boto:Headers: {} 2022-07-18T03:06:37.989 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:37.989 DEBUG:boto:Port: 8001 2022-07-18T03:06:37.990 DEBUG:boto:Params: {} 2022-07-18T03:06:37.990 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:06:37.991 DEBUG:boto:Token: None 2022-07-18T03:06:37.991 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:37 GMT /mlvank-3/ 2022-07-18T03:06:37.992 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:SNuz1+edsLQqpI/uYA8Mx6iDw5A= 2022-07-18T03:06:37.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:37 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:SNuz1+edsLQqpI/uYA8Mx6iDw5A=', 'Content-Length': '0'} 2022-07-18T03:06:38.035 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000894abff46fd7f28b-0062d4ce3d-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:38 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:38.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:38.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:38.037 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:38.090 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:38.090 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:38.143 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:06:38.157 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:06:38.158 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:38.158 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:38.159 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:06:38.159 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:38.159 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:38.159 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:38.160 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:38.160 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:38.160 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:38.160 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:38.161 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:38.161 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:38.161 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:38.161 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113598.017351_95.1", 2022-07-18T03:06:38.162 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:38.017351Z" 2022-07-18T03:06:38.162 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:38.162 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:06:38.164 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:38.164 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:06:38.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:38.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:38.167 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:38.227 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:38.227 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:38.281 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:38.282 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:38.283 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:38.284 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:38.284 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:38.285 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:38.285 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:38.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:38.286 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:38.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:38.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:38.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:38.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:38.289 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:38.289 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:38.290 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:38.291 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:38.292 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:38.292 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:38.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:38.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:38.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:38.293 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:38.294 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:38.294 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:38.294 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:38.295 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:38.295 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:38.295 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:38.295 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:38.296 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:38.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:38.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:38.299 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:38.299 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:38.299 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:38.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:38.301 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113595.414466_84.1", 2022-07-18T03:06:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:35.414466Z", 2022-07-18T03:06:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:38Z" 2022-07-18T03:06:38.305 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:38.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113595.414466_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:35.414466Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:38Z" } 2022-07-18T03:06:38.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113595.414466_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:35.414466Z', 'realm_epoch': 2}}] 2022-07-18T03:06:38.307 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:38.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113595.414466_84.1'} 2022-07-18T03:06:38.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113598.017351_95.1 target=1_1658113595.414466_84.1 2022-07-18T03:06:43.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:43.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:43.311 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:43.369 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:43.370 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:43.428 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:43.428 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:43.429 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:43.429 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:43.430 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:43.430 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:43.431 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.432 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:43.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.433 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:43.434 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.434 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.437 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:43.437 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.438 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:43.438 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.439 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:43.439 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.440 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.441 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:43.446 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.446 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.447 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:43.447 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:43.447 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.448 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:43.448 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.449 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:43.449 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.449 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.450 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:43.450 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.450 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:43.450 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.451 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:43.452 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.454 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:43.454 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.454 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.455 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113598.017351_95.1", 2022-07-18T03:06:43.455 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.455 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:43.455 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.456 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:38.017351Z", 2022-07-18T03:06:43.456 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.457 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:43.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.457 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:43.458 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:43.458 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:43.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.459 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:43Z" 2022-07-18T03:06:43.459 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:43.460 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113598.017351_95.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:38.017351Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:43Z" } 2022-07-18T03:06:43.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113598.017351_95.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:38.017351Z', 'realm_epoch': 2}}] 2022-07-18T03:06:43.462 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:43.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:43.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-07-18T03:06:43.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-07-18T03:06:43.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:43.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:43.465 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:43.521 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:43.521 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:43.598 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:06:43.599 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": { 2022-07-18T03:06:43.600 INFO:teuthology.orchestra.run.smithi133.stdout: "info": { 2022-07-18T03:06:43.600 INFO:teuthology.orchestra.run.smithi133.stdout: "status": "sync", 2022-07-18T03:06:43.601 INFO:teuthology.orchestra.run.smithi133.stdout: "num_shards": 4, 2022-07-18T03:06:43.601 INFO:teuthology.orchestra.run.smithi133.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:43.602 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:43.602 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:43.603 INFO:teuthology.orchestra.run.smithi133.stdout: "markers": [ 2022-07-18T03:06:43.603 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:43.604 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 0, 2022-07-18T03:06:43.604 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:43.605 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:43.605 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:43.606 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:43.606 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 1, 2022-07-18T03:06:43.607 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:43.607 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:43.608 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:43.608 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:43.609 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:43.609 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:43.610 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 1, 2022-07-18T03:06:43.611 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:43.611 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:43.612 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:43.612 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:43.613 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:43.613 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:43.614 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:43.614 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:43.615 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:43.615 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:43.616 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:43.616 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 2, 2022-07-18T03:06:43.617 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:43.617 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:43.618 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:43.619 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:43.619 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:43.619 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:43.620 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:43.620 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:43.620 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:43.621 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:43.621 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:43.621 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 3, 2022-07-18T03:06:43.621 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:43.622 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:43.622 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113598.017351_95.1", 2022-07-18T03:06:43.622 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:43.622 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:43.623 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:43.623 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:06:38.017351Z", 2022-07-18T03:06:43.623 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:43.623 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:43.624 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:43.624 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:06:43.624 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:43.624 INFO:teuthology.orchestra.run.smithi133.stdout: "full_sync": { 2022-07-18T03:06:43.625 INFO:teuthology.orchestra.run.smithi133.stdout: "total": 1, 2022-07-18T03:06:43.625 INFO:teuthology.orchestra.run.smithi133.stdout: "complete": 1 2022-07-18T03:06:43.625 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:43.626 INFO:teuthology.orchestra.run.smithi133.stdout: "current_time": "2022-07-18T03:06:43Z" 2022-07-18T03:06:43.626 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:06:43.628 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113598.017351_95.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:38.017351Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:43Z" } 2022-07-18T03:06:43.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113598.017351_95.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:38.017351Z', 'realm_epoch': 2}}] 2022-07-18T03:06:43.629 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:43.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:43.630 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-07-18T03:06:43.631 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-07-18T03:06:43.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:43.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:43.632 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:43.683 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:43.684 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:43.748 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:43.748 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:43.749 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:43.749 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:43.749 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:43.751 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:43.752 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.753 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:43.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.754 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:43.754 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.755 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.755 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:43.756 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:43.757 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.757 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:43.758 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.760 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:43.760 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.761 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.761 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:43.762 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.762 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:43.763 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.763 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:43.766 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.766 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.768 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:43.768 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.769 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.769 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:43.770 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.770 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:43.770 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.771 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:43.771 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:43.773 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:43.774 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:43.774 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:43.774 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113598.017351_95.1", 2022-07-18T03:06:43.775 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:43.775 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:43.776 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:43.776 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:38.017351Z", 2022-07-18T03:06:43.777 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:43.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:43.778 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:43.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.778 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:43.779 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:43.779 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:43.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:43.780 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:43Z" 2022-07-18T03:06:43.780 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:43.782 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113598.017351_95.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:38.017351Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:43Z" } 2022-07-18T03:06:43.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113598.017351_95.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:38.017351Z', 'realm_epoch': 2}}] 2022-07-18T03:06:43.783 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:43.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113598.017351_95.1'} 2022-07-18T03:06:43.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-07-18T03:06:43.785 DEBUG:boto:path=/mlvank-1/ 2022-07-18T03:06:43.786 DEBUG:boto:auth_path=/mlvank-1/ 2022-07-18T03:06:43.787 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.788 DEBUG:boto:Path: /mlvank-1/ 2022-07-18T03:06:43.789 DEBUG:boto:Data: 2022-07-18T03:06:43.790 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.791 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:43.791 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.793 DEBUG:boto:Params: {} 2022-07-18T03:06:43.794 DEBUG:boto:Token: None 2022-07-18T03:06:43.795 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-1/ 2022-07-18T03:06:43.796 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:SCcY0Urklpwk6dQjClN/EP/6azQ= 2022-07-18T03:06:43.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:SCcY0Urklpwk6dQjClN/EP/6azQ=', 'Content-Length': '0'} 2022-07-18T03:06:43.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d41a2dd2b50dc99-0062d4ce43-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.802 DEBUG:boto:path=/mlvank-2/ 2022-07-18T03:06:43.803 DEBUG:boto:auth_path=/mlvank-2/ 2022-07-18T03:06:43.804 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.805 DEBUG:boto:Path: /mlvank-2/ 2022-07-18T03:06:43.806 DEBUG:boto:Data: 2022-07-18T03:06:43.806 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.807 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:43.808 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.808 DEBUG:boto:Params: {} 2022-07-18T03:06:43.809 DEBUG:boto:Token: None 2022-07-18T03:06:43.810 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-2/ 2022-07-18T03:06:43.810 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:iphHJutQycs8566P/AJysBen51A= 2022-07-18T03:06:43.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:iphHJutQycs8566P/AJysBen51A=', 'Content-Length': '0'} 2022-07-18T03:06:43.814 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002589fa3b74ef7180-0062d4ce43-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.814 DEBUG:boto:path=/mlvank-3/ 2022-07-18T03:06:43.815 DEBUG:boto:auth_path=/mlvank-3/ 2022-07-18T03:06:43.816 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.817 DEBUG:boto:Path: /mlvank-3/ 2022-07-18T03:06:43.817 DEBUG:boto:Data: 2022-07-18T03:06:43.818 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.818 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:43.819 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.820 DEBUG:boto:Params: {} 2022-07-18T03:06:43.821 DEBUG:boto:Token: None 2022-07-18T03:06:43.821 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-3/ 2022-07-18T03:06:43.823 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:ANKFgkMMytXk2tnKj5SAc2u1OBk= 2022-07-18T03:06:43.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:ANKFgkMMytXk2tnKj5SAc2u1OBk=', 'Content-Length': '0'} 2022-07-18T03:06:43.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea30f481bf48b449-0062d4ce43-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.828 DEBUG:boto:path=/mlvank-1/ 2022-07-18T03:06:43.828 DEBUG:boto:auth_path=/mlvank-1/ 2022-07-18T03:06:43.829 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.830 DEBUG:boto:Path: /mlvank-1/ 2022-07-18T03:06:43.831 DEBUG:boto:Data: 2022-07-18T03:06:43.832 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.833 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:43.833 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.834 DEBUG:boto:Params: {} 2022-07-18T03:06:43.835 DEBUG:boto:Token: None 2022-07-18T03:06:43.837 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-1/ 2022-07-18T03:06:43.838 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:SCcY0Urklpwk6dQjClN/EP/6azQ= 2022-07-18T03:06:43.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:SCcY0Urklpwk6dQjClN/EP/6azQ=', 'Content-Length': '0'} 2022-07-18T03:06:43.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e17c1aafafa3599-0062d4ce43-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.842 DEBUG:boto:path=/mlvank-2/ 2022-07-18T03:06:43.843 DEBUG:boto:auth_path=/mlvank-2/ 2022-07-18T03:06:43.845 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.846 DEBUG:boto:Path: /mlvank-2/ 2022-07-18T03:06:43.847 DEBUG:boto:Data: 2022-07-18T03:06:43.847 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.848 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:43.849 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.850 DEBUG:boto:Params: {} 2022-07-18T03:06:43.851 DEBUG:boto:Token: None 2022-07-18T03:06:43.852 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-2/ 2022-07-18T03:06:43.853 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:iphHJutQycs8566P/AJysBen51A= 2022-07-18T03:06:43.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:iphHJutQycs8566P/AJysBen51A=', 'Content-Length': '0'} 2022-07-18T03:06:43.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de4d62f4dc056e55-0062d4ce43-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.858 DEBUG:boto:path=/mlvank-3/ 2022-07-18T03:06:43.859 DEBUG:boto:auth_path=/mlvank-3/ 2022-07-18T03:06:43.860 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.861 DEBUG:boto:Path: /mlvank-3/ 2022-07-18T03:06:43.862 DEBUG:boto:Data: 2022-07-18T03:06:43.863 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.864 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:43.864 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.865 DEBUG:boto:Params: {} 2022-07-18T03:06:43.866 DEBUG:boto:Token: None 2022-07-18T03:06:43.867 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-3/ 2022-07-18T03:06:43.868 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:ANKFgkMMytXk2tnKj5SAc2u1OBk= 2022-07-18T03:06:43.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:ANKFgkMMytXk2tnKj5SAc2u1OBk=', 'Content-Length': '0'} 2022-07-18T03:06:43.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beee5a1b2904ef7c-0062d4ce43-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.874 DEBUG:boto:path=/mlvank-1/ 2022-07-18T03:06:43.875 DEBUG:boto:auth_path=/mlvank-1/ 2022-07-18T03:06:43.875 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.876 DEBUG:boto:Path: /mlvank-1/ 2022-07-18T03:06:43.877 DEBUG:boto:Data: 2022-07-18T03:06:43.877 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.878 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:43.878 DEBUG:boto:Port: 8001 2022-07-18T03:06:43.879 DEBUG:boto:Params: {} 2022-07-18T03:06:43.880 DEBUG:boto:Token: None 2022-07-18T03:06:43.881 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-1/ 2022-07-18T03:06:43.882 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:SCcY0Urklpwk6dQjClN/EP/6azQ= 2022-07-18T03:06:43.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:SCcY0Urklpwk6dQjClN/EP/6azQ=', 'Content-Length': '0'} 2022-07-18T03:06:43.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a79c7b2b0645548b-0062d4ce43-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.888 DEBUG:boto:path=/mlvank-2/ 2022-07-18T03:06:43.889 DEBUG:boto:auth_path=/mlvank-2/ 2022-07-18T03:06:43.890 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.891 DEBUG:boto:Path: /mlvank-2/ 2022-07-18T03:06:43.891 DEBUG:boto:Data: 2022-07-18T03:06:43.892 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.893 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:43.895 DEBUG:boto:Port: 8001 2022-07-18T03:06:43.895 DEBUG:boto:Params: {} 2022-07-18T03:06:43.896 DEBUG:boto:Token: None 2022-07-18T03:06:43.896 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-2/ 2022-07-18T03:06:43.897 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:iphHJutQycs8566P/AJysBen51A= 2022-07-18T03:06:43.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:iphHJutQycs8566P/AJysBen51A=', 'Content-Length': '0'} 2022-07-18T03:06:43.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008820af8c1349b9a4-0062d4ce43-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.902 DEBUG:boto:path=/mlvank-3/ 2022-07-18T03:06:43.903 DEBUG:boto:auth_path=/mlvank-3/ 2022-07-18T03:06:43.904 DEBUG:boto:Method: HEAD 2022-07-18T03:06:43.905 DEBUG:boto:Path: /mlvank-3/ 2022-07-18T03:06:43.906 DEBUG:boto:Data: 2022-07-18T03:06:43.907 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.908 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:43.908 DEBUG:boto:Port: 8001 2022-07-18T03:06:43.909 DEBUG:boto:Params: {} 2022-07-18T03:06:43.909 DEBUG:boto:Token: None 2022-07-18T03:06:43.910 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:43 GMT /mlvank-3/ 2022-07-18T03:06:43.911 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:ANKFgkMMytXk2tnKj5SAc2u1OBk= 2022-07-18T03:06:43.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:ANKFgkMMytXk2tnKj5SAc2u1OBk=', 'Content-Length': '0'} 2022-07-18T03:06:43.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b90cbfb703cf6b6-0062d4ce43-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2022-07-18T03:06:43.930 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-4 2022-07-18T03:06:43.931 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:43.932 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:43.932 DEBUG:boto:Method: PUT 2022-07-18T03:06:43.933 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:43.933 DEBUG:boto:Data: 2022-07-18T03:06:43.934 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.934 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:43.935 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.935 DEBUG:boto:Params: {} 2022-07-18T03:06:43.936 DEBUG:boto:Token: None 2022-07-18T03:06:43.937 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:43 GMT /mlvank-4/ 2022-07-18T03:06:43.937 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QQuGBMnyV98dZuqSX8fZOm1WBGc= 2022-07-18T03:06:43.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QQuGBMnyV98dZuqSX8fZOm1WBGc=', 'Content-Length': '0'} 2022-07-18T03:06:43.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ea2a7bfaedbd09e2-0062d4ce43-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:43.965 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-5 2022-07-18T03:06:43.965 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:43.966 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:43.967 DEBUG:boto:Method: PUT 2022-07-18T03:06:43.967 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:43.968 DEBUG:boto:Data: 2022-07-18T03:06:43.968 DEBUG:boto:Headers: {} 2022-07-18T03:06:43.968 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:43.969 DEBUG:boto:Port: 8000 2022-07-18T03:06:43.969 DEBUG:boto:Params: {} 2022-07-18T03:06:43.970 DEBUG:boto:Token: None 2022-07-18T03:06:43.971 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:43 GMT /mlvank-5/ 2022-07-18T03:06:43.971 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:BlpkDTMWCQiu4A8Au+1iLYqZQQk= 2022-07-18T03:06:43.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:43 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:BlpkDTMWCQiu4A8Au+1iLYqZQQk=', 'Content-Length': '0'} 2022-07-18T03:06:44.003 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000039a4af296f3d9ceb-0062d4ce43-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.005 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-6 2022-07-18T03:06:44.006 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.007 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.008 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.009 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.010 DEBUG:boto:Data: 2022-07-18T03:06:44.011 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.012 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.013 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.014 DEBUG:boto:Params: {} 2022-07-18T03:06:44.015 DEBUG:boto:Token: None 2022-07-18T03:06:44.016 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.017 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo= 2022-07-18T03:06:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo=', 'Content-Length': '0'} 2022-07-18T03:06:44.042 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dadfa4f036103004-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.046 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:44.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:44.095 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:44.143 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:06:44.154 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:06:44.155 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.155 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:44.155 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:06:44.156 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.156 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.156 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:44.156 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:44.157 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.157 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.157 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:44.158 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:44.158 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.158 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.158 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:44.159 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:44.032700Z" 2022-07-18T03:06:44.159 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:44.159 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:06:44.160 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.161 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:06:44.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.164 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:44.220 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:44.220 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:44.276 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:44.277 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:44.277 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:44.278 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:44.278 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:44.278 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:44.279 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.280 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:44.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:44.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.281 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.281 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:44.282 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.282 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:44.282 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:44.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.284 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:44.284 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.284 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.285 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:44.285 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.285 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:44.285 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.286 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:44.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.286 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:44.287 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.287 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:44.288 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:44.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:44.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.291 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:44.291 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.291 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.292 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:44.292 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.292 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:44.292 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.293 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:44.032700Z", 2022-07-18T03:06:44.293 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.294 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:44.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.295 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:44.296 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:44.296 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:44.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.297 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:44Z" 2022-07-18T03:06:44.297 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:44.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113604.032700_107.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:44.032700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:44Z" } 2022-07-18T03:06:44.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113604.032700_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:44.032700Z', 'realm_epoch': 2}}] 2022-07-18T03:06:44.301 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.302 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.303 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-07-18T03:06:44.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-07-18T03:06:44.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.307 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:44.356 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:44.357 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:44.431 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:06:44.432 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": { 2022-07-18T03:06:44.432 INFO:teuthology.orchestra.run.smithi133.stdout: "info": { 2022-07-18T03:06:44.433 INFO:teuthology.orchestra.run.smithi133.stdout: "status": "sync", 2022-07-18T03:06:44.433 INFO:teuthology.orchestra.run.smithi133.stdout: "num_shards": 4, 2022-07-18T03:06:44.434 INFO:teuthology.orchestra.run.smithi133.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:44.434 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:44.435 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.435 INFO:teuthology.orchestra.run.smithi133.stdout: "markers": [ 2022-07-18T03:06:44.436 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.436 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 0, 2022-07-18T03:06:44.437 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:44.437 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:44.438 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:44.439 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:44.439 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 1, 2022-07-18T03:06:44.440 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:44.440 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:44.441 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:44.441 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:44.442 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.442 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.443 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 1, 2022-07-18T03:06:44.443 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:44.444 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:44.444 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:44.445 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:44.446 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:44.446 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:44.447 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:44.447 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:44.448 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:44.448 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.449 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.449 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 2, 2022-07-18T03:06:44.450 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:44.451 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:44.451 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:44.452 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:44.452 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:44.453 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:44.453 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:44.454 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:44.454 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:44.455 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.455 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:44.456 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 3, 2022-07-18T03:06:44.456 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:44.457 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:44.457 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:44.458 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:44.459 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:44.459 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:44.460 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:06:44.032700Z", 2022-07-18T03:06:44.460 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:44.461 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:44.461 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:44.462 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:06:44.462 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.463 INFO:teuthology.orchestra.run.smithi133.stdout: "full_sync": { 2022-07-18T03:06:44.463 INFO:teuthology.orchestra.run.smithi133.stdout: "total": 1, 2022-07-18T03:06:44.464 INFO:teuthology.orchestra.run.smithi133.stdout: "complete": 1 2022-07-18T03:06:44.464 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:44.465 INFO:teuthology.orchestra.run.smithi133.stdout: "current_time": "2022-07-18T03:06:44Z" 2022-07-18T03:06:44.465 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:06:44.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113604.032700_107.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:44.032700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:44Z" } 2022-07-18T03:06:44.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113604.032700_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:44.032700Z', 'realm_epoch': 2}}] 2022-07-18T03:06:44.470 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.472 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-07-18T03:06:44.473 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-07-18T03:06:44.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:44.477 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:44.529 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:44.530 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:44.589 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:44.589 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:44.590 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:44.590 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:44.591 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:44.591 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:44.592 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.593 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:44.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.594 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:44.595 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.595 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.596 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:44.598 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.599 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:44.599 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.600 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:44.600 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.601 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.602 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:44.603 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.603 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.604 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:44.604 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.607 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:44.607 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.607 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.608 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:44.608 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.609 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:44.609 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.610 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:44.610 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:44.612 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:44.612 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:44.613 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:44.613 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:44.614 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:44.614 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:44.614 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:44.615 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:44.032700Z", 2022-07-18T03:06:44.615 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:44.615 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.616 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:44.616 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:44.617 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.617 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:44.618 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:44.618 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:44.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:44.619 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:44Z" 2022-07-18T03:06:44.619 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:44.621 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113604.032700_107.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:44.032700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:44Z" } 2022-07-18T03:06:44.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113604.032700_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:44.032700Z', 'realm_epoch': 2}}] 2022-07-18T03:06:44.622 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:44.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-07-18T03:06:44.624 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:44.624 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:44.625 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.625 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:44.626 DEBUG:boto:Data: 2022-07-18T03:06:44.626 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.627 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:44.628 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.628 DEBUG:boto:Params: {} 2022-07-18T03:06:44.629 DEBUG:boto:Token: None 2022-07-18T03:06:44.630 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-4/ 2022-07-18T03:06:44.631 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:xtVCwDdvSqPfwCRE6IlWn7gu7Vg= 2022-07-18T03:06:44.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:xtVCwDdvSqPfwCRE6IlWn7gu7Vg=', 'Content-Length': '0'} 2022-07-18T03:06:44.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008105f5fb589ba06-0062d4ce44-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.636 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:44.637 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:44.637 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.638 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:44.639 DEBUG:boto:Data: 2022-07-18T03:06:44.640 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.641 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:44.642 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.643 DEBUG:boto:Params: {} 2022-07-18T03:06:44.644 DEBUG:boto:Token: None 2022-07-18T03:06:44.645 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-5/ 2022-07-18T03:06:44.646 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:OPmQoAlz0XUZQ/p67tf59vdfbFs= 2022-07-18T03:06:44.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:OPmQoAlz0XUZQ/p67tf59vdfbFs=', 'Content-Length': '0'} 2022-07-18T03:06:44.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000865ef245ed6e2f9b-0062d4ce44-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.651 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.652 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.652 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.653 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.653 DEBUG:boto:Data: 2022-07-18T03:06:44.654 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.654 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:44.655 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.655 DEBUG:boto:Params: {} 2022-07-18T03:06:44.656 DEBUG:boto:Token: None 2022-07-18T03:06:44.656 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.657 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:UeLcRVtl9Ma29RrMYTIWJSkvID4= 2022-07-18T03:06:44.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:UeLcRVtl9Ma29RrMYTIWJSkvID4=', 'Content-Length': '0'} 2022-07-18T03:06:44.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085a8ff179d6517ef-0062d4ce44-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.661 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:44.661 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:44.662 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.662 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:44.663 DEBUG:boto:Data: 2022-07-18T03:06:44.664 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.664 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:44.665 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.665 DEBUG:boto:Params: {} 2022-07-18T03:06:44.666 DEBUG:boto:Token: None 2022-07-18T03:06:44.666 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-4/ 2022-07-18T03:06:44.667 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:xtVCwDdvSqPfwCRE6IlWn7gu7Vg= 2022-07-18T03:06:44.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:xtVCwDdvSqPfwCRE6IlWn7gu7Vg=', 'Content-Length': '0'} 2022-07-18T03:06:44.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c73d489873e06270-0062d4ce44-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.671 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:44.672 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:44.672 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.673 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:44.673 DEBUG:boto:Data: 2022-07-18T03:06:44.674 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.674 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:44.675 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.676 DEBUG:boto:Params: {} 2022-07-18T03:06:44.676 DEBUG:boto:Token: None 2022-07-18T03:06:44.677 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-5/ 2022-07-18T03:06:44.678 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:OPmQoAlz0XUZQ/p67tf59vdfbFs= 2022-07-18T03:06:44.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:OPmQoAlz0XUZQ/p67tf59vdfbFs=', 'Content-Length': '0'} 2022-07-18T03:06:44.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d1f91217264674b-0062d4ce44-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.683 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.684 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.685 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.686 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.690 DEBUG:boto:Data: 2022-07-18T03:06:44.691 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.692 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:44.693 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.694 DEBUG:boto:Params: {} 2022-07-18T03:06:44.695 DEBUG:boto:Token: None 2022-07-18T03:06:44.696 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.697 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:UeLcRVtl9Ma29RrMYTIWJSkvID4= 2022-07-18T03:06:44.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:UeLcRVtl9Ma29RrMYTIWJSkvID4=', 'Content-Length': '0'} 2022-07-18T03:06:44.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f83cfe4df7c73d4f-0062d4ce44-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.702 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:44.703 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:44.704 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.705 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:44.706 DEBUG:boto:Data: 2022-07-18T03:06:44.707 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.708 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.708 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.709 DEBUG:boto:Params: {} 2022-07-18T03:06:44.710 DEBUG:boto:Token: None 2022-07-18T03:06:44.711 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-4/ 2022-07-18T03:06:44.712 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:xtVCwDdvSqPfwCRE6IlWn7gu7Vg= 2022-07-18T03:06:44.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:xtVCwDdvSqPfwCRE6IlWn7gu7Vg=', 'Content-Length': '0'} 2022-07-18T03:06:44.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003494e14fe255ca5b-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.720 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:44.722 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:44.723 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.724 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:44.725 DEBUG:boto:Data: 2022-07-18T03:06:44.725 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.726 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.727 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.728 DEBUG:boto:Params: {} 2022-07-18T03:06:44.729 DEBUG:boto:Token: None 2022-07-18T03:06:44.730 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-5/ 2022-07-18T03:06:44.730 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:OPmQoAlz0XUZQ/p67tf59vdfbFs= 2022-07-18T03:06:44.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:OPmQoAlz0XUZQ/p67tf59vdfbFs=', 'Content-Length': '0'} 2022-07-18T03:06:44.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee49200f4c8ef989-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.735 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.736 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.736 DEBUG:boto:Method: HEAD 2022-07-18T03:06:44.737 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.738 DEBUG:boto:Data: 2022-07-18T03:06:44.738 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.739 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.739 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.740 DEBUG:boto:Params: {} 2022-07-18T03:06:44.741 DEBUG:boto:Token: None 2022-07-18T03:06:44.742 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.743 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:UeLcRVtl9Ma29RrMYTIWJSkvID4= 2022-07-18T03:06:44.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:UeLcRVtl9Ma29RrMYTIWJSkvID4=', 'Content-Length': '0'} 2022-07-18T03:06:44.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000061341dd1c4568c3-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.748 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:44.748 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:44.749 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.749 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:44.750 DEBUG:boto:Data: 2022-07-18T03:06:44.751 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.751 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:44.752 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.752 DEBUG:boto:Params: {} 2022-07-18T03:06:44.753 DEBUG:boto:Token: None 2022-07-18T03:06:44.754 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-4/ 2022-07-18T03:06:44.754 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:t/NptauxzAV44kHC/ax7EULBrhU= 2022-07-18T03:06:44.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:t/NptauxzAV44kHC/ax7EULBrhU=', 'Content-Length': '0'} 2022-07-18T03:06:44.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009996190e343812c6-0062d4ce44-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.758 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:44.759 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:44.759 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.760 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:44.760 DEBUG:boto:Data: 2022-07-18T03:06:44.761 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.761 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:44.762 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.763 DEBUG:boto:Params: {} 2022-07-18T03:06:44.763 DEBUG:boto:Token: None 2022-07-18T03:06:44.764 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-5/ 2022-07-18T03:06:44.765 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:vuvHXlDsu4VSGYulnLSRnYoolbc= 2022-07-18T03:06:44.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:vuvHXlDsu4VSGYulnLSRnYoolbc=', 'Content-Length': '0'} 2022-07-18T03:06:44.768 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000083197458ebe73d18-0062d4ce44-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.769 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.769 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.770 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.770 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.771 DEBUG:boto:Data: 2022-07-18T03:06:44.771 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.772 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:44.772 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.773 DEBUG:boto:Params: {} 2022-07-18T03:06:44.773 DEBUG:boto:Token: None 2022-07-18T03:06:44.774 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.774 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo= 2022-07-18T03:06:44.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo=', 'Content-Length': '0'} 2022-07-18T03:06:44.778 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000022923a34d20e576e-0062d4ce44-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.779 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:44.780 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:44.780 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.781 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:44.781 DEBUG:boto:Data: 2022-07-18T03:06:44.782 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.782 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:44.783 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.783 DEBUG:boto:Params: {} 2022-07-18T03:06:44.784 DEBUG:boto:Token: None 2022-07-18T03:06:44.785 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-4/ 2022-07-18T03:06:44.785 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:t/NptauxzAV44kHC/ax7EULBrhU= 2022-07-18T03:06:44.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:t/NptauxzAV44kHC/ax7EULBrhU=', 'Content-Length': '0'} 2022-07-18T03:06:44.792 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000aa332b99621e2ae-0062d4ce44-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.793 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:44.793 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:44.795 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.796 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:44.797 DEBUG:boto:Data: 2022-07-18T03:06:44.797 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.798 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:44.799 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.800 DEBUG:boto:Params: {} 2022-07-18T03:06:44.801 DEBUG:boto:Token: None 2022-07-18T03:06:44.802 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-5/ 2022-07-18T03:06:44.804 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:vuvHXlDsu4VSGYulnLSRnYoolbc= 2022-07-18T03:06:44.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:vuvHXlDsu4VSGYulnLSRnYoolbc=', 'Content-Length': '0'} 2022-07-18T03:06:44.851 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000178807b7e945bcf9-0062d4ce44-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.853 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.854 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.855 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.856 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.856 DEBUG:boto:Data: 2022-07-18T03:06:44.857 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.857 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:44.858 DEBUG:boto:Port: 8000 2022-07-18T03:06:44.858 DEBUG:boto:Params: {} 2022-07-18T03:06:44.859 DEBUG:boto:Token: None 2022-07-18T03:06:44.860 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.861 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo= 2022-07-18T03:06:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo=', 'Content-Length': '0'} 2022-07-18T03:06:44.909 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000948e75f3578c16cf-0062d4ce44-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.909 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:44.910 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:44.910 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.911 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:44.911 DEBUG:boto:Data: 2022-07-18T03:06:44.912 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.912 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.913 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.913 DEBUG:boto:Params: {} 2022-07-18T03:06:44.914 DEBUG:boto:Token: None 2022-07-18T03:06:44.915 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-4/ 2022-07-18T03:06:44.916 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:t/NptauxzAV44kHC/ax7EULBrhU= 2022-07-18T03:06:44.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:t/NptauxzAV44kHC/ax7EULBrhU=', 'Content-Length': '0'} 2022-07-18T03:06:44.924 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000612b5d454745328-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.924 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:44.925 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:44.925 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.926 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:44.927 DEBUG:boto:Data: 2022-07-18T03:06:44.928 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.928 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.929 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.929 DEBUG:boto:Params: {} 2022-07-18T03:06:44.930 DEBUG:boto:Token: None 2022-07-18T03:06:44.931 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-5/ 2022-07-18T03:06:44.931 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:vuvHXlDsu4VSGYulnLSRnYoolbc= 2022-07-18T03:06:44.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:vuvHXlDsu4VSGYulnLSRnYoolbc=', 'Content-Length': '0'} 2022-07-18T03:06:44.978 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e04e76377d30091c-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:44.979 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:44.980 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:44.980 DEBUG:boto:Method: PUT 2022-07-18T03:06:44.981 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:44.982 DEBUG:boto:Data: 2022-07-18T03:06:44.983 DEBUG:boto:Headers: {} 2022-07-18T03:06:44.983 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:44.984 DEBUG:boto:Port: 8001 2022-07-18T03:06:44.984 DEBUG:boto:Params: {} 2022-07-18T03:06:44.985 DEBUG:boto:Token: None 2022-07-18T03:06:44.986 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:44 GMT /mlvank-6/ 2022-07-18T03:06:44.986 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo= 2022-07-18T03:06:44.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:44 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:6T3BITnK9gN1S0KyVduBQ7k4xPo=', 'Content-Length': '0'} 2022-07-18T03:06:45.033 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000155941c45874e7d9-0062d4ce44-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.035 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:45.035 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:45.036 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.037 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:45.038 DEBUG:boto:Data: 2022-07-18T03:06:45.039 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.040 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.042 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.043 DEBUG:boto:Params: {} 2022-07-18T03:06:45.044 DEBUG:boto:Token: None 2022-07-18T03:06:45.045 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-4/ 2022-07-18T03:06:45.046 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE= 2022-07-18T03:06:45.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE=', 'Content-Length': '0'} 2022-07-18T03:06:45.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000711431f8856a5790-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.051 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:45.052 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:45.052 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.053 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:45.053 DEBUG:boto:Data: 2022-07-18T03:06:45.054 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.054 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.055 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.056 DEBUG:boto:Params: {} 2022-07-18T03:06:45.058 DEBUG:boto:Token: None 2022-07-18T03:06:45.059 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-5/ 2022-07-18T03:06:45.060 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY= 2022-07-18T03:06:45.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY=', 'Content-Length': '0'} 2022-07-18T03:06:45.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020033c9e68f503cd-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.064 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:45.065 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:45.066 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.067 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:45.067 DEBUG:boto:Data: 2022-07-18T03:06:45.068 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.068 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.069 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.069 DEBUG:boto:Params: {} 2022-07-18T03:06:45.070 DEBUG:boto:Token: None 2022-07-18T03:06:45.070 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-6/ 2022-07-18T03:06:45.071 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo= 2022-07-18T03:06:45.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo=', 'Content-Length': '0'} 2022-07-18T03:06:45.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c42163f047ce0d0a-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.075 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:45.075 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:45.076 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.076 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:45.077 DEBUG:boto:Data: 2022-07-18T03:06:45.077 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.078 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.078 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.079 DEBUG:boto:Params: {} 2022-07-18T03:06:45.079 DEBUG:boto:Token: None 2022-07-18T03:06:45.080 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-4/ 2022-07-18T03:06:45.081 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE= 2022-07-18T03:06:45.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE=', 'Content-Length': '0'} 2022-07-18T03:06:45.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002546734ed35f29b8-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.085 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:45.085 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:45.086 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.086 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:45.087 DEBUG:boto:Data: 2022-07-18T03:06:45.087 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.088 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.088 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.089 DEBUG:boto:Params: {} 2022-07-18T03:06:45.089 DEBUG:boto:Token: None 2022-07-18T03:06:45.090 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-5/ 2022-07-18T03:06:45.091 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY= 2022-07-18T03:06:45.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY=', 'Content-Length': '0'} 2022-07-18T03:06:45.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096bbcb3bab4ad980-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.095 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:45.096 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:45.096 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.097 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:45.098 DEBUG:boto:Data: 2022-07-18T03:06:45.099 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.099 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.100 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.101 DEBUG:boto:Params: {} 2022-07-18T03:06:45.103 DEBUG:boto:Token: None 2022-07-18T03:06:45.104 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-6/ 2022-07-18T03:06:45.104 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo= 2022-07-18T03:06:45.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo=', 'Content-Length': '0'} 2022-07-18T03:06:45.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beb004a9b14727ed-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.109 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:45.110 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:45.111 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.112 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:45.112 DEBUG:boto:Data: 2022-07-18T03:06:45.113 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.113 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.114 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.114 DEBUG:boto:Params: {} 2022-07-18T03:06:45.115 DEBUG:boto:Token: None 2022-07-18T03:06:45.116 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-4/ 2022-07-18T03:06:45.117 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE= 2022-07-18T03:06:45.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE=', 'Content-Length': '0'} 2022-07-18T03:06:45.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091eb36908aec187c-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.121 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:45.122 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:45.122 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.123 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:45.123 DEBUG:boto:Data: 2022-07-18T03:06:45.124 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.124 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.125 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.125 DEBUG:boto:Params: {} 2022-07-18T03:06:45.126 DEBUG:boto:Token: None 2022-07-18T03:06:45.127 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-5/ 2022-07-18T03:06:45.127 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY= 2022-07-18T03:06:45.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY=', 'Content-Length': '0'} 2022-07-18T03:06:45.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6548e40f93f2a00-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.131 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:45.131 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:45.132 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.132 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:45.133 DEBUG:boto:Data: 2022-07-18T03:06:45.133 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.134 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.135 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.136 DEBUG:boto:Params: {} 2022-07-18T03:06:45.137 DEBUG:boto:Token: None 2022-07-18T03:06:45.138 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-6/ 2022-07-18T03:06:45.138 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo= 2022-07-18T03:06:45.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo=', 'Content-Length': '0'} 2022-07-18T03:06:45.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5d7a0b0c7917bd1-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.145 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:45.194 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:45.194 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:45.243 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:06:45.266 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:06:45.266 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.267 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:45.267 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:06:45.267 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.267 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.268 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:45.268 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:45.268 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.268 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.269 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:45.269 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:45.269 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.269 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.270 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:45.270 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:44.032700Z" 2022-07-18T03:06:45.270 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:45.270 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:06:45.273 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.273 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:06:45.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.275 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:45.324 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:45.324 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:45.387 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:45.388 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:45.388 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:45.389 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:45.389 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:45.390 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:45.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.391 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:45.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:45.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:45.394 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.394 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:45.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.400 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:45.401 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.401 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:45.402 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.403 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:45.403 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.403 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.404 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:45.404 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.404 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:45.404 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.405 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:44.032700Z", 2022-07-18T03:06:45.405 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.405 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.405 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.406 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:45.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.406 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:45.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:45.407 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:45.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.407 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:45Z" 2022-07-18T03:06:45.407 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:45.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113604.032700_107.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:44.032700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:45Z" } 2022-07-18T03:06:45.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113604.032700_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:44.032700Z', 'realm_epoch': 2}}] 2022-07-18T03:06:45.415 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.417 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-07-18T03:06:45.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-07-18T03:06:45.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.421 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:45.471 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:45.472 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:45.521 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:06:45.521 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": { 2022-07-18T03:06:45.521 INFO:teuthology.orchestra.run.smithi133.stdout: "info": { 2022-07-18T03:06:45.522 INFO:teuthology.orchestra.run.smithi133.stdout: "status": "sync", 2022-07-18T03:06:45.522 INFO:teuthology.orchestra.run.smithi133.stdout: "num_shards": 4, 2022-07-18T03:06:45.522 INFO:teuthology.orchestra.run.smithi133.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:45.523 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:45.523 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.523 INFO:teuthology.orchestra.run.smithi133.stdout: "markers": [ 2022-07-18T03:06:45.523 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.524 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 0, 2022-07-18T03:06:45.524 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:45.524 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:45.525 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:45.525 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:45.525 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 1, 2022-07-18T03:06:45.525 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:45.526 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:45.527 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:45.527 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:45.527 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.528 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.528 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 1, 2022-07-18T03:06:45.528 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:45.528 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:45.529 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:45.529 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:45.529 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:45.530 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:45.530 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:45.530 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:45.531 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:45.531 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.531 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.531 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 2, 2022-07-18T03:06:45.532 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:45.532 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:45.532 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:45.532 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:45.533 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:45.533 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:45.533 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:45.533 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:45.534 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:45.534 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.534 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:45.534 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 3, 2022-07-18T03:06:45.535 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:45.535 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:45.535 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:45.536 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:45.536 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:45.536 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:45.536 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:06:44.032700Z", 2022-07-18T03:06:45.536 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:45.537 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:45.537 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:45.538 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:06:45.538 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.539 INFO:teuthology.orchestra.run.smithi133.stdout: "full_sync": { 2022-07-18T03:06:45.539 INFO:teuthology.orchestra.run.smithi133.stdout: "total": 1, 2022-07-18T03:06:45.539 INFO:teuthology.orchestra.run.smithi133.stdout: "complete": 1 2022-07-18T03:06:45.539 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:45.540 INFO:teuthology.orchestra.run.smithi133.stdout: "current_time": "2022-07-18T03:06:45Z" 2022-07-18T03:06:45.540 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:06:45.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113604.032700_107.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:44.032700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:45Z" } 2022-07-18T03:06:45.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113604.032700_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:44.032700Z', 'realm_epoch': 2}}] 2022-07-18T03:06:45.543 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-07-18T03:06:45.544 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-07-18T03:06:45.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.546 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:45.597 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:45.598 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:45.651 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:45.652 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:45.652 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:45.652 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:45.652 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:45.653 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:45.653 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.653 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:45.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.654 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:45.654 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.655 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.655 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:45.655 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.656 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:45.656 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.656 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:45.656 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.657 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:45.658 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.658 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.658 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:45.658 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.659 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:45.659 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.659 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:45.660 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.660 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.661 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:45.661 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.661 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.662 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:45.662 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.662 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:45.663 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.663 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:45.663 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:45.664 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:45.664 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:45.665 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:45.665 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113604.032700_107.1", 2022-07-18T03:06:45.665 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:45.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:45.666 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:45.666 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:44.032700Z", 2022-07-18T03:06:45.666 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:45.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:45.667 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:45.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.668 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:45.668 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:45.668 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:45.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:45.669 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:45Z" 2022-07-18T03:06:45.669 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:45.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113604.032700_107.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:44.032700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:45Z" } 2022-07-18T03:06:45.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113604.032700_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:44.032700Z', 'realm_epoch': 2}}] 2022-07-18T03:06:45.672 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113604.032700_107.1'} 2022-07-18T03:06:45.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-07-18T03:06:45.673 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:45.674 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:45.674 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.675 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:45.675 DEBUG:boto:Data: 2022-07-18T03:06:45.676 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.677 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.677 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.678 DEBUG:boto:Params: {} 2022-07-18T03:06:45.678 DEBUG:boto:Token: None 2022-07-18T03:06:45.679 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-4/ 2022-07-18T03:06:45.680 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE= 2022-07-18T03:06:45.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE=', 'Content-Length': '0'} 2022-07-18T03:06:45.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000970546a05aacf258-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.683 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:45.684 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:45.684 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.685 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:45.686 DEBUG:boto:Data: 2022-07-18T03:06:45.686 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.687 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.687 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.688 DEBUG:boto:Params: {} 2022-07-18T03:06:45.688 DEBUG:boto:Token: None 2022-07-18T03:06:45.689 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-5/ 2022-07-18T03:06:45.690 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY= 2022-07-18T03:06:45.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY=', 'Content-Length': '0'} 2022-07-18T03:06:45.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df41b3b8a7f913f8-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.693 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:45.694 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:45.694 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.695 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:45.695 DEBUG:boto:Data: 2022-07-18T03:06:45.696 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.696 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.697 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.697 DEBUG:boto:Params: {} 2022-07-18T03:06:45.698 DEBUG:boto:Token: None 2022-07-18T03:06:45.698 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-6/ 2022-07-18T03:06:45.699 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo= 2022-07-18T03:06:45.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo=', 'Content-Length': '0'} 2022-07-18T03:06:45.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013bf89aeb8d2af7e-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.702 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:45.703 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:45.703 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.704 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:45.705 DEBUG:boto:Data: 2022-07-18T03:06:45.705 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.706 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.706 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.707 DEBUG:boto:Params: {} 2022-07-18T03:06:45.707 DEBUG:boto:Token: None 2022-07-18T03:06:45.708 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-4/ 2022-07-18T03:06:45.709 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE= 2022-07-18T03:06:45.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE=', 'Content-Length': '0'} 2022-07-18T03:06:45.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3f3b3903958a5a3-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.713 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:45.713 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:45.714 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.714 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:45.715 DEBUG:boto:Data: 2022-07-18T03:06:45.715 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.716 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.716 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.717 DEBUG:boto:Params: {} 2022-07-18T03:06:45.718 DEBUG:boto:Token: None 2022-07-18T03:06:45.718 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-5/ 2022-07-18T03:06:45.719 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY= 2022-07-18T03:06:45.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY=', 'Content-Length': '0'} 2022-07-18T03:06:45.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024dc73d7f701ad8b-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.723 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:45.724 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:45.724 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.725 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:45.725 DEBUG:boto:Data: 2022-07-18T03:06:45.726 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.727 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.727 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.728 DEBUG:boto:Params: {} 2022-07-18T03:06:45.728 DEBUG:boto:Token: None 2022-07-18T03:06:45.729 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-6/ 2022-07-18T03:06:45.729 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo= 2022-07-18T03:06:45.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo=', 'Content-Length': '0'} 2022-07-18T03:06:45.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da72b22c5657b6ec-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.733 DEBUG:boto:path=/mlvank-4/ 2022-07-18T03:06:45.734 DEBUG:boto:auth_path=/mlvank-4/ 2022-07-18T03:06:45.734 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.735 DEBUG:boto:Path: /mlvank-4/ 2022-07-18T03:06:45.736 DEBUG:boto:Data: 2022-07-18T03:06:45.736 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.737 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.738 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.738 DEBUG:boto:Params: {} 2022-07-18T03:06:45.739 DEBUG:boto:Token: None 2022-07-18T03:06:45.740 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-4/ 2022-07-18T03:06:45.740 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE= 2022-07-18T03:06:45.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:3Wpyd1ZZMaXzBil5wDCY5g8QcFE=', 'Content-Length': '0'} 2022-07-18T03:06:45.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058d3879ceb8bec17-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.744 DEBUG:boto:path=/mlvank-5/ 2022-07-18T03:06:45.744 DEBUG:boto:auth_path=/mlvank-5/ 2022-07-18T03:06:45.745 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.745 DEBUG:boto:Path: /mlvank-5/ 2022-07-18T03:06:45.746 DEBUG:boto:Data: 2022-07-18T03:06:45.746 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.747 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.747 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.748 DEBUG:boto:Params: {} 2022-07-18T03:06:45.749 DEBUG:boto:Token: None 2022-07-18T03:06:45.749 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-5/ 2022-07-18T03:06:45.750 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY= 2022-07-18T03:06:45.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qY8rHaO/F0k/b6579XpD2r+BBCY=', 'Content-Length': '0'} 2022-07-18T03:06:45.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f9c44deda6bd099-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.755 DEBUG:boto:path=/mlvank-6/ 2022-07-18T03:06:45.756 DEBUG:boto:auth_path=/mlvank-6/ 2022-07-18T03:06:45.757 DEBUG:boto:Method: HEAD 2022-07-18T03:06:45.758 DEBUG:boto:Path: /mlvank-6/ 2022-07-18T03:06:45.758 DEBUG:boto:Data: 2022-07-18T03:06:45.759 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.760 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.762 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.762 DEBUG:boto:Params: {} 2022-07-18T03:06:45.764 DEBUG:boto:Token: None 2022-07-18T03:06:45.765 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:45 GMT /mlvank-6/ 2022-07-18T03:06:45.765 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo= 2022-07-18T03:06:45.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:XsmpbdnBoFkJupQ2xPRPdbeNPeo=', 'Content-Length': '0'} 2022-07-18T03:06:45.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a25f5765b94ae794-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2022-07-18T03:06:45.771 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-7 2022-07-18T03:06:45.771 DEBUG:boto:path=/mlvank-7/ 2022-07-18T03:06:45.772 DEBUG:boto:auth_path=/mlvank-7/ 2022-07-18T03:06:45.772 DEBUG:boto:Method: PUT 2022-07-18T03:06:45.773 DEBUG:boto:Path: /mlvank-7/ 2022-07-18T03:06:45.773 DEBUG:boto:Data: 2022-07-18T03:06:45.774 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.774 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:45.775 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.775 DEBUG:boto:Params: {} 2022-07-18T03:06:45.776 DEBUG:boto:Token: None 2022-07-18T03:06:45.776 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:45 GMT /mlvank-7/ 2022-07-18T03:06:45.777 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:bJMVd9lAzTBGdMaS2rvf6E0RX90= 2022-07-18T03:06:45.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:bJMVd9lAzTBGdMaS2rvf6E0RX90=', 'Content-Length': '0'} 2022-07-18T03:06:45.791 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b74d58f590e420ac-0062d4ce45-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.792 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-8 2022-07-18T03:06:45.792 DEBUG:boto:path=/mlvank-8/ 2022-07-18T03:06:45.793 DEBUG:boto:auth_path=/mlvank-8/ 2022-07-18T03:06:45.794 DEBUG:boto:Method: PUT 2022-07-18T03:06:45.795 DEBUG:boto:Path: /mlvank-8/ 2022-07-18T03:06:45.795 DEBUG:boto:Data: 2022-07-18T03:06:45.796 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.797 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:45.798 DEBUG:boto:Port: 8000 2022-07-18T03:06:45.799 DEBUG:boto:Params: {} 2022-07-18T03:06:45.800 DEBUG:boto:Token: None 2022-07-18T03:06:45.801 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:45 GMT /mlvank-8/ 2022-07-18T03:06:45.801 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:HGnTyfStErmtAiFTjezF4/uDeKw= 2022-07-18T03:06:45.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:HGnTyfStErmtAiFTjezF4/uDeKw=', 'Content-Length': '0'} 2022-07-18T03:06:45.840 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000971eb33364958df0-0062d4ce45-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.841 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-9 2022-07-18T03:06:45.841 DEBUG:boto:path=/mlvank-9/ 2022-07-18T03:06:45.842 DEBUG:boto:auth_path=/mlvank-9/ 2022-07-18T03:06:45.842 DEBUG:boto:Method: PUT 2022-07-18T03:06:45.843 DEBUG:boto:Path: /mlvank-9/ 2022-07-18T03:06:45.843 DEBUG:boto:Data: 2022-07-18T03:06:45.844 DEBUG:boto:Headers: {} 2022-07-18T03:06:45.844 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:45.845 DEBUG:boto:Port: 8001 2022-07-18T03:06:45.845 DEBUG:boto:Params: {} 2022-07-18T03:06:45.846 DEBUG:boto:Token: None 2022-07-18T03:06:45.847 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:06:45 GMT /mlvank-9/ 2022-07-18T03:06:45.847 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Mxvj+FM4xAXiwM5b4bL3Hmrf+Do= 2022-07-18T03:06:45.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:45 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Mxvj+FM4xAXiwM5b4bL3Hmrf+Do=', 'Content-Length': '0'} 2022-07-18T03:06:45.893 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb80ea612b3778cc-0062d4ce45-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:45.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:45.895 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:45.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:45.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:46.012 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:06:46.035 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:06:46.035 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.036 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:46.036 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:06:46.037 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.037 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.037 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:46.038 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:46.038 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.039 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.039 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:46.040 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:46.040 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.041 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.041 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113605.873980_119.1", 2022-07-18T03:06:46.042 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:45.873980Z" 2022-07-18T03:06:46.042 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.043 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:06:46.044 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:06:46.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.046 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:46.094 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:46.094 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:46.169 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:46.170 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:46.170 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:46.171 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:46.171 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:46.171 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:46.171 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.172 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:46.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.172 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:46.173 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.173 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.173 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:46.173 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.174 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:46.174 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.174 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:46.175 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.176 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:46.176 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.176 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.176 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:46.177 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.177 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:46.177 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.177 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:46.178 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.179 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:46.179 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.179 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.180 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:46.180 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.180 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:46.180 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.181 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:46.181 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.183 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:46.184 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.184 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.185 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113605.873980_119.1", 2022-07-18T03:06:46.185 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.185 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:46.186 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.186 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:45.873980Z", 2022-07-18T03:06:46.187 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.188 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.189 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:46.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.190 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:46.190 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:46.191 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:46.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.192 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:46Z" 2022-07-18T03:06:46.192 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:46.195 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113605.873980_119.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:45.873980Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:46Z" } 2022-07-18T03:06:46.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113605.873980_119.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:45.873980Z', 'realm_epoch': 2}}] 2022-07-18T03:06:46.197 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.198 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-07-18T03:06:46.199 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-07-18T03:06:46.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.202 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:46.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:46.251 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:46.302 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2022-07-18T03:06:46.303 INFO:teuthology.orchestra.run.smithi133.stdout: "sync_status": { 2022-07-18T03:06:46.304 INFO:teuthology.orchestra.run.smithi133.stdout: "info": { 2022-07-18T03:06:46.304 INFO:teuthology.orchestra.run.smithi133.stdout: "status": "sync", 2022-07-18T03:06:46.304 INFO:teuthology.orchestra.run.smithi133.stdout: "num_shards": 4, 2022-07-18T03:06:46.304 INFO:teuthology.orchestra.run.smithi133.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:46.305 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:46.305 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.305 INFO:teuthology.orchestra.run.smithi133.stdout: "markers": [ 2022-07-18T03:06:46.305 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.306 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 0, 2022-07-18T03:06:46.306 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:46.306 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:46.306 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:46.307 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:46.307 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 1, 2022-07-18T03:06:46.307 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:46.307 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:46.308 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:46.308 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.308 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.308 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.309 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 1, 2022-07-18T03:06:46.309 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:46.309 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:46.309 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:46.310 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:46.310 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:46.310 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:46.310 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:46.311 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:46.311 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.311 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.311 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.312 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 2, 2022-07-18T03:06:46.312 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:46.312 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:46.312 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:46.313 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:46.313 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:46.313 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:46.313 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "0.000000", 2022-07-18T03:06:46.314 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:46.314 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.315 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.315 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.316 INFO:teuthology.orchestra.run.smithi133.stdout: "key": 3, 2022-07-18T03:06:46.316 INFO:teuthology.orchestra.run.smithi133.stdout: "val": { 2022-07-18T03:06:46.316 INFO:teuthology.orchestra.run.smithi133.stdout: "state": 1, 2022-07-18T03:06:46.316 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113605.873980_119.1", 2022-07-18T03:06:46.317 INFO:teuthology.orchestra.run.smithi133.stdout: "next_step_marker": "", 2022-07-18T03:06:46.317 INFO:teuthology.orchestra.run.smithi133.stdout: "total_entries": 0, 2022-07-18T03:06:46.317 INFO:teuthology.orchestra.run.smithi133.stdout: "pos": 0, 2022-07-18T03:06:46.317 INFO:teuthology.orchestra.run.smithi133.stdout: "timestamp": "2022-07-18T03:06:45.873980Z", 2022-07-18T03:06:46.318 INFO:teuthology.orchestra.run.smithi133.stdout: "realm_epoch": 2 2022-07-18T03:06:46.318 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.318 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.319 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2022-07-18T03:06:46.319 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.319 INFO:teuthology.orchestra.run.smithi133.stdout: "full_sync": { 2022-07-18T03:06:46.319 INFO:teuthology.orchestra.run.smithi133.stdout: "total": 1, 2022-07-18T03:06:46.320 INFO:teuthology.orchestra.run.smithi133.stdout: "complete": 1 2022-07-18T03:06:46.320 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.320 INFO:teuthology.orchestra.run.smithi133.stdout: "current_time": "2022-07-18T03:06:46Z" 2022-07-18T03:06:46.320 INFO:teuthology.orchestra.run.smithi133.stdout:} 2022-07-18T03:06:46.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113605.873980_119.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:45.873980Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:46Z" } 2022-07-18T03:06:46.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113605.873980_119.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:45.873980Z', 'realm_epoch': 2}}] 2022-07-18T03:06:46.322 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.323 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-07-18T03:06:46.323 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-07-18T03:06:46.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.325 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:46.373 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:46.374 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:46.428 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:46.428 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:46.428 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:46.429 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:46.429 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:46.429 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:46.429 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.430 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:46.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.431 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:46.432 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.432 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.433 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:46.433 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.434 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:46.434 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.435 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:46.435 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.436 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.437 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:46.437 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.438 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.438 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:46.438 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.439 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:46.439 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.439 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:46.439 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.440 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.441 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:46.441 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.441 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.442 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:46.442 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.442 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:46.443 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.443 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:46.444 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.445 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:46.446 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:46.447 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:46.447 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:46.448 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113605.873980_119.1", 2022-07-18T03:06:46.448 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:46.449 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:46.450 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:46.450 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:45.873980Z", 2022-07-18T03:06:46.451 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:46.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:46.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:46.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.453 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:46.454 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:46.455 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:46.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:46.456 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:46Z" 2022-07-18T03:06:46.457 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:46.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113605.873980_119.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:45.873980Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:46Z" } 2022-07-18T03:06:46.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113605.873980_119.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:45.873980Z', 'realm_epoch': 2}}] 2022-07-18T03:06:46.460 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.460 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113605.873980_119.1'} 2022-07-18T03:06:46.461 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-07-18T03:06:46.461 DEBUG:boto:path=/mlvank-7/ 2022-07-18T03:06:46.462 DEBUG:boto:auth_path=/mlvank-7/ 2022-07-18T03:06:46.462 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.463 DEBUG:boto:Path: /mlvank-7/ 2022-07-18T03:06:46.463 DEBUG:boto:Data: 2022-07-18T03:06:46.464 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.464 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:46.465 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.465 DEBUG:boto:Params: {} 2022-07-18T03:06:46.466 DEBUG:boto:Token: None 2022-07-18T03:06:46.467 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-7/ 2022-07-18T03:06:46.467 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Wy78nYn0zRtTp7RCMwxq7g7CSVY= 2022-07-18T03:06:46.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Wy78nYn0zRtTp7RCMwxq7g7CSVY=', 'Content-Length': '0'} 2022-07-18T03:06:46.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ad1abbd8b80079f-0062d4ce46-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.471 DEBUG:boto:path=/mlvank-8/ 2022-07-18T03:06:46.472 DEBUG:boto:auth_path=/mlvank-8/ 2022-07-18T03:06:46.472 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.473 DEBUG:boto:Path: /mlvank-8/ 2022-07-18T03:06:46.473 DEBUG:boto:Data: 2022-07-18T03:06:46.474 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.474 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:46.475 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.475 DEBUG:boto:Params: {} 2022-07-18T03:06:46.476 DEBUG:boto:Token: None 2022-07-18T03:06:46.477 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-8/ 2022-07-18T03:06:46.477 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:9MBgq5rJog8x2VIv0i+zF+IJoVM= 2022-07-18T03:06:46.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:9MBgq5rJog8x2VIv0i+zF+IJoVM=', 'Content-Length': '0'} 2022-07-18T03:06:46.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029588f1ec4d6a805-0062d4ce46-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.481 DEBUG:boto:path=/mlvank-9/ 2022-07-18T03:06:46.482 DEBUG:boto:auth_path=/mlvank-9/ 2022-07-18T03:06:46.482 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.483 DEBUG:boto:Path: /mlvank-9/ 2022-07-18T03:06:46.483 DEBUG:boto:Data: 2022-07-18T03:06:46.484 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.484 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:46.485 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.485 DEBUG:boto:Params: {} 2022-07-18T03:06:46.486 DEBUG:boto:Token: None 2022-07-18T03:06:46.486 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-9/ 2022-07-18T03:06:46.487 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:C2mXYi9ELBE6oxQU8SfWXhRC3Mc= 2022-07-18T03:06:46.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:C2mXYi9ELBE6oxQU8SfWXhRC3Mc=', 'Content-Length': '0'} 2022-07-18T03:06:46.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029cf3980a0540ed3-0062d4ce46-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.491 DEBUG:boto:path=/mlvank-7/ 2022-07-18T03:06:46.491 DEBUG:boto:auth_path=/mlvank-7/ 2022-07-18T03:06:46.492 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.492 DEBUG:boto:Path: /mlvank-7/ 2022-07-18T03:06:46.493 DEBUG:boto:Data: 2022-07-18T03:06:46.493 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.494 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:46.494 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.495 DEBUG:boto:Params: {} 2022-07-18T03:06:46.495 DEBUG:boto:Token: None 2022-07-18T03:06:46.496 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-7/ 2022-07-18T03:06:46.496 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Wy78nYn0zRtTp7RCMwxq7g7CSVY= 2022-07-18T03:06:46.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Wy78nYn0zRtTp7RCMwxq7g7CSVY=', 'Content-Length': '0'} 2022-07-18T03:06:46.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113bd64e99f67184-0062d4ce46-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.501 DEBUG:boto:path=/mlvank-8/ 2022-07-18T03:06:46.502 DEBUG:boto:auth_path=/mlvank-8/ 2022-07-18T03:06:46.503 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.504 DEBUG:boto:Path: /mlvank-8/ 2022-07-18T03:06:46.505 DEBUG:boto:Data: 2022-07-18T03:06:46.505 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.506 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:46.507 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.508 DEBUG:boto:Params: {} 2022-07-18T03:06:46.509 DEBUG:boto:Token: None 2022-07-18T03:06:46.510 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-8/ 2022-07-18T03:06:46.511 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:9MBgq5rJog8x2VIv0i+zF+IJoVM= 2022-07-18T03:06:46.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:9MBgq5rJog8x2VIv0i+zF+IJoVM=', 'Content-Length': '0'} 2022-07-18T03:06:46.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c83f482144371a6d-0062d4ce46-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.514 DEBUG:boto:path=/mlvank-9/ 2022-07-18T03:06:46.515 DEBUG:boto:auth_path=/mlvank-9/ 2022-07-18T03:06:46.516 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.516 DEBUG:boto:Path: /mlvank-9/ 2022-07-18T03:06:46.516 DEBUG:boto:Data: 2022-07-18T03:06:46.517 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.517 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:46.518 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.519 DEBUG:boto:Params: {} 2022-07-18T03:06:46.519 DEBUG:boto:Token: None 2022-07-18T03:06:46.520 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-9/ 2022-07-18T03:06:46.520 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:C2mXYi9ELBE6oxQU8SfWXhRC3Mc= 2022-07-18T03:06:46.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:C2mXYi9ELBE6oxQU8SfWXhRC3Mc=', 'Content-Length': '0'} 2022-07-18T03:06:46.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009269a1b1e049352d-0062d4ce46-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.524 DEBUG:boto:path=/mlvank-7/ 2022-07-18T03:06:46.524 DEBUG:boto:auth_path=/mlvank-7/ 2022-07-18T03:06:46.525 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.525 DEBUG:boto:Path: /mlvank-7/ 2022-07-18T03:06:46.526 DEBUG:boto:Data: 2022-07-18T03:06:46.526 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.527 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:46.527 DEBUG:boto:Port: 8001 2022-07-18T03:06:46.528 DEBUG:boto:Params: {} 2022-07-18T03:06:46.529 DEBUG:boto:Token: None 2022-07-18T03:06:46.529 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-7/ 2022-07-18T03:06:46.530 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Wy78nYn0zRtTp7RCMwxq7g7CSVY= 2022-07-18T03:06:46.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Wy78nYn0zRtTp7RCMwxq7g7CSVY=', 'Content-Length': '0'} 2022-07-18T03:06:46.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000152191b64944d994-0062d4ce46-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.534 DEBUG:boto:path=/mlvank-8/ 2022-07-18T03:06:46.534 DEBUG:boto:auth_path=/mlvank-8/ 2022-07-18T03:06:46.535 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.535 DEBUG:boto:Path: /mlvank-8/ 2022-07-18T03:06:46.536 DEBUG:boto:Data: 2022-07-18T03:06:46.536 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.537 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:46.537 DEBUG:boto:Port: 8001 2022-07-18T03:06:46.538 DEBUG:boto:Params: {} 2022-07-18T03:06:46.539 DEBUG:boto:Token: None 2022-07-18T03:06:46.540 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-8/ 2022-07-18T03:06:46.541 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:9MBgq5rJog8x2VIv0i+zF+IJoVM= 2022-07-18T03:06:46.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:9MBgq5rJog8x2VIv0i+zF+IJoVM=', 'Content-Length': '0'} 2022-07-18T03:06:46.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d97b0526d135e1d7-0062d4ce46-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.544 DEBUG:boto:path=/mlvank-9/ 2022-07-18T03:06:46.545 DEBUG:boto:auth_path=/mlvank-9/ 2022-07-18T03:06:46.545 DEBUG:boto:Method: HEAD 2022-07-18T03:06:46.546 DEBUG:boto:Path: /mlvank-9/ 2022-07-18T03:06:46.547 DEBUG:boto:Data: 2022-07-18T03:06:46.548 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.548 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:46.549 DEBUG:boto:Port: 8001 2022-07-18T03:06:46.549 DEBUG:boto:Params: {} 2022-07-18T03:06:46.550 DEBUG:boto:Token: None 2022-07-18T03:06:46.550 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:06:46 GMT /mlvank-9/ 2022-07-18T03:06:46.551 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:C2mXYi9ELBE6oxQU8SfWXhRC3Mc= 2022-07-18T03:06:46.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:C2mXYi9ELBE6oxQU8SfWXhRC3Mc=', 'Content-Length': '0'} 2022-07-18T03:06:46.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054559a6baec54452-0062d4ce46-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.555 DEBUG:boto:path=/mlvank-7/ 2022-07-18T03:06:46.556 DEBUG:boto:auth_path=/mlvank-7/ 2022-07-18T03:06:46.557 DEBUG:boto:Method: DELETE 2022-07-18T03:06:46.557 DEBUG:boto:Path: /mlvank-7/ 2022-07-18T03:06:46.558 DEBUG:boto:Data: 2022-07-18T03:06:46.558 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.559 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:06:46.559 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.560 DEBUG:boto:Params: {} 2022-07-18T03:06:46.561 DEBUG:boto:Token: None 2022-07-18T03:06:46.561 DEBUG:boto:StringToSign: DELETE Mon, 18 Jul 2022 03:06:46 GMT /mlvank-7/ 2022-07-18T03:06:46.562 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:x6C8IX6iTU5tRZ/X5O3Uh4Uqlr8= 2022-07-18T03:06:46.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:x6C8IX6iTU5tRZ/X5O3Uh4Uqlr8=', 'Content-Length': '0'} 2022-07-18T03:06:46.589 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000510ba3f162435209-0062d4ce46-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.590 DEBUG:boto:path=/mlvank-8/ 2022-07-18T03:06:46.591 DEBUG:boto:auth_path=/mlvank-8/ 2022-07-18T03:06:46.591 DEBUG:boto:Method: DELETE 2022-07-18T03:06:46.592 DEBUG:boto:Path: /mlvank-8/ 2022-07-18T03:06:46.592 DEBUG:boto:Data: 2022-07-18T03:06:46.593 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.593 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:06:46.594 DEBUG:boto:Port: 8000 2022-07-18T03:06:46.594 DEBUG:boto:Params: {} 2022-07-18T03:06:46.595 DEBUG:boto:Token: None 2022-07-18T03:06:46.595 DEBUG:boto:StringToSign: DELETE Mon, 18 Jul 2022 03:06:46 GMT /mlvank-8/ 2022-07-18T03:06:46.596 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:j0dZ0j+CMU924oql3YSwhln/PYA= 2022-07-18T03:06:46.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:j0dZ0j+CMU924oql3YSwhln/PYA=', 'Content-Length': '0'} 2022-07-18T03:06:46.672 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052a770b2131304d4-0062d4ce46-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.673 DEBUG:boto:path=/mlvank-9/ 2022-07-18T03:06:46.674 DEBUG:boto:auth_path=/mlvank-9/ 2022-07-18T03:06:46.674 DEBUG:boto:Method: DELETE 2022-07-18T03:06:46.675 DEBUG:boto:Path: /mlvank-9/ 2022-07-18T03:06:46.676 DEBUG:boto:Data: 2022-07-18T03:06:46.677 DEBUG:boto:Headers: {} 2022-07-18T03:06:46.678 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:06:46.679 DEBUG:boto:Port: 8001 2022-07-18T03:06:46.680 DEBUG:boto:Params: {} 2022-07-18T03:06:46.681 DEBUG:boto:Token: None 2022-07-18T03:06:46.682 DEBUG:boto:StringToSign: DELETE Mon, 18 Jul 2022 03:06:46 GMT /mlvank-9/ 2022-07-18T03:06:46.683 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:A4bdTNzJgn6QCmWZuU+7SnBHYsM= 2022-07-18T03:06:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:06:46 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:A4bdTNzJgn6QCmWZuU+7SnBHYsM=', 'Content-Length': '0'} 2022-07-18T03:06:46.766 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086128fd4c0f9a5fd-0062d4ce46-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:06:46.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.768 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:46.816 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:46.816 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:46.867 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:06:46.879 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:06:46.879 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.880 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:46.880 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:06:46.880 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.881 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.881 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:46.881 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:46.881 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.882 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.882 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:06:46.882 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:06:46.882 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:06:46.883 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:06:46.883 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:06:46.883 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:06:46.883 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:06:46.884 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:06:46.885 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:06:46.885 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:06:46.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:46.887 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:46.939 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:46.939 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:47.063 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:47.064 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:47.064 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:47.064 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:47.065 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:47.065 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:47.065 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:47.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:47.066 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:47.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:47.066 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:47.067 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:47.067 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:47.068 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:47.068 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:47.068 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:47.069 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:47.069 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:47.069 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:47.069 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:47.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:47.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:47.070 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:47.071 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:47.071 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:47.071 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:47.071 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:47.072 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:47.072 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:47.073 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:47.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:47.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:47.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:47.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:47.076 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:47.076 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:47.077 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:47.077 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:47.078 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:47.078 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:47.079 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:47.079 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:47.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:47.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:47.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:47.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:47.082 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:47.083 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:47.083 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:47.084 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:06:47.084 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:47.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:06:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:47.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:47.087 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:47.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:47.088 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:47.089 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:47.089 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:47.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:47.090 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:47Z" 2022-07-18T03:06:47.091 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:47.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:47Z" } 2022-07-18T03:06:47.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:06:47.093 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:06:47.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:06:47.094 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:06:52.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:52.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:52.097 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:52.146 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:52.147 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:52.222 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:52.223 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:52.226 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:52.227 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:52.227 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:52.228 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:52.230 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:52.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:52.232 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:52.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:52.233 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:52.233 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:52.233 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:52.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:52.234 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:52.235 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:52.235 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:52.236 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:52.236 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:52.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:52.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:52.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:52.242 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:52.243 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:52.243 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:52.244 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:52.244 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:52.245 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:52.245 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:52.246 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:52.247 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:52.247 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:52.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:52.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:52.249 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:52.249 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:52.250 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:52.250 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:52.251 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:52.251 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:52.252 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:52.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:52.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:52.256 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:52.257 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:06:52.257 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:52.258 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:52.258 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:52.259 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:06:52.259 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:52.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:52.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:52.261 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:52.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:52.262 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:52.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:52Z" 2022-07-18T03:06:52.265 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:52.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:52Z" } 2022-07-18T03:06:52.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:06:52.267 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:06:52.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:06:52.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:06:57.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:57.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:06:57.272 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:06:57.328 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:06:57.329 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:06:57.399 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:06:57.399 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:06:57.400 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:06:57.400 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:06:57.400 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:06:57.401 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:06:57.401 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:57.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:57.402 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:06:57.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:57.403 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:06:57.403 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:57.404 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:57.404 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:06:57.405 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:57.405 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:06:57.406 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:57.406 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:06:57.407 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:57.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:57.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:57.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:57.408 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:06:57.408 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:57.409 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:57.409 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:57.409 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:57.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:57.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:57.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:57.411 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:06:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:06:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:57.413 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:57.413 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:06:57.413 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:06:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:06:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:06:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:06:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:06:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:06:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:06:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:06:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:06:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:06:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:06:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:06:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:06:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:06:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:06:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:06:57.419 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:06:57Z" 2022-07-18T03:06:57.419 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:06:57.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:06:57Z" } 2022-07-18T03:06:57.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:06:57.422 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:06:57.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:06:57.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:02.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:02.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:02.427 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:02.477 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:02.478 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:02.529 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:02.529 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:02.529 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:02.530 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:02.530 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:02.530 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:02.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:02.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:02.531 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:02.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:02.532 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:02.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:02.532 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:02.532 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:02.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:02.533 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:02.533 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:02.533 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:02.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:02.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:02.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:02.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:02.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:02.535 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:02.535 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:02.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:02.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:02.536 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:02.536 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:02.537 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:02.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:02.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:02.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:02.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:02.538 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:02.539 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:02.540 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:02.540 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:02.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:02.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:02.541 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:02.541 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:02.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:02.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:02.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:02.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:02.542 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:02.542 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:02.543 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:02.543 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:02.543 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:02.544 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:02.544 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:02.544 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:02.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:02.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:02.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:02.545 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:02.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:02.546 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:02.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:02.546 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:02.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:02.547 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:02Z" 2022-07-18T03:07:02.547 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:02.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:02Z" } 2022-07-18T03:07:02.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:02.550 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:02.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:02.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:07.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:07.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:07.553 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:07.602 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:07.602 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:07.670 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:07.671 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:07.671 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:07.671 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:07.671 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:07.672 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:07.672 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:07.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:07.672 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:07.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:07.673 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:07.673 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:07.673 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:07.674 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:07.674 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:07.674 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:07.674 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:07.675 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:07.675 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:07.675 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:07.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:07.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:07.676 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:07.676 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:07.676 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:07.677 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:07.677 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:07.677 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:07.678 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:07.678 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:07.678 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:07.678 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:07.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:07.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:07.679 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:07.679 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:07.680 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:07.680 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:07.680 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:07.680 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:07.681 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:07.681 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:07.681 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:07.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:07.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:07.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:07.682 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:07.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:07.683 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:07.683 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:07.683 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:07.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:07.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:07.684 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:07.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:07.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:07.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:07.685 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:07.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:07.686 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:07.686 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:07.687 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:07.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:07.687 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:07Z" 2022-07-18T03:07:07.687 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:07.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:07Z" } 2022-07-18T03:07:07.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:07.690 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:07.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:07.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:12.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:12.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:12.694 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:12.744 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:12.744 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:12.812 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:12.812 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:12.813 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:12.813 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:12.813 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:12.813 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:12.814 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:12.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:12.814 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:12.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:12.815 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:12.815 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:12.815 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:12.816 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:12.816 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:12.816 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:12.816 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:12.816 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:12.817 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:12.817 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:12.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:12.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:12.818 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:12.818 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:12.818 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:12.818 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:12.819 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:12.819 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:12.819 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:12.819 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:12.820 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:12.820 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:12.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:12.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:12.822 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:12.822 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:12.822 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:12.822 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:12.823 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:12.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:12.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:12.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:12.824 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:12.824 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:12.825 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:12.825 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:12.825 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:12.826 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:12.826 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:12.826 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:12.826 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:12.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:12.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:12.827 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:12.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:12.828 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:12.828 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:12.828 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:12.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:12.829 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:12Z" 2022-07-18T03:07:12.829 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:12.831 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:12Z" } 2022-07-18T03:07:12.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:12.832 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:12.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:12.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:17.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:17.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:17.836 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:17.892 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:17.892 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:17.952 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:17.953 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:17.954 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:17.954 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:17.955 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:17.955 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:17.956 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:17.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:17.956 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:17.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:17.957 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:17.957 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:17.958 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:17.958 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:17.959 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:17.959 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:17.959 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:17.960 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:17.960 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:17.960 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:17.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:17.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:17.961 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:17.961 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:17.962 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:17.962 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:17.962 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:17.962 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:17.963 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:17.963 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:17.963 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:17.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:17.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:17.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:17.964 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:17.965 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:17.965 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:17.965 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:17.965 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:17.966 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:17.966 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:17.966 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:17.966 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:17.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:17.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:17.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:17.968 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:17.968 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:17.968 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:17.968 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:17.969 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:17.969 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:17.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:17.970 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:17.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:17.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:17.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:17.971 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:17.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:17.971 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:17.971 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:17.972 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:17.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:17.972 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:17Z" 2022-07-18T03:07:17.973 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:17.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:17Z" } 2022-07-18T03:07:17.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:17.975 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:17.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:17.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:22.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:22.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:22.981 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:23.030 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:23.031 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:23.106 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:23.106 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:23.106 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:23.107 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:23.107 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:23.107 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:23.107 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:23.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:23.108 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:23.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:23.108 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:23.109 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:23.109 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:23.109 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:23.110 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:23.110 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:23.110 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:23.110 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:23.111 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:23.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:23.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:23.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:23.112 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:23.112 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:23.112 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:23.113 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:23.113 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:23.113 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:23.113 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:23.114 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:23.114 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:23.114 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:23.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:23.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:23.115 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:23.116 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:23.116 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:23.116 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:23.117 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:23.117 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:23.118 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:23.119 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:23.119 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:23.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:23.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:23.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:23.122 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:23.122 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:23.123 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:23.124 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:23.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:23.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:23.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:23.125 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:23.127 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:23.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:23.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:23.128 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:23.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:23.128 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:23.128 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:23.129 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:23.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:23.129 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:23Z" 2022-07-18T03:07:23.129 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:23.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:23Z" } 2022-07-18T03:07:23.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:23.133 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:23.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:23.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:28.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:28.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:28.138 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:28.187 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:28.188 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:28.263 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:28Z" 2022-07-18T03:07:28.282 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:28.283 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:28Z" } 2022-07-18T03:07:28.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:28.284 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:28.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:28.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:33.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:33.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:33.289 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:33.359 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:33.359 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:33.436 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:33.436 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:33.437 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:33.437 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:33.438 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:33.438 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:33.438 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:33.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:33.439 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:33.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:33.439 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:33.440 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:33.440 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:33.440 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:33.440 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:33.441 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:33.441 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:33.441 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:33.442 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:33.442 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:33.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:33.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:33.443 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:33.443 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:33.443 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:33.443 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:33.444 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:33.444 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:33.444 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:33.445 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:33.445 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:33.445 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:33.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:33.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:33.446 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:33.446 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:33.447 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:33.447 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:33.447 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:33.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:33.448 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:33.448 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:33.448 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:33.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:33.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:33.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:33.449 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:33.449 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:33.450 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:33.450 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:33.450 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:33.450 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:33.451 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:33.451 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:33.451 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:33.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:33.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:33.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:33.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:33.452 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:33.453 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:33.453 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:33.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:33.453 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:33Z" 2022-07-18T03:07:33.454 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:33.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:33Z" } 2022-07-18T03:07:33.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:33.456 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:33.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:33.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:38.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:38.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:38.460 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:38.508 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:38.509 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:38.579 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:38.579 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:38.580 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:38.580 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:38.580 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:38.581 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:38.581 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:38.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:38.581 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:38.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:38.582 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:38.582 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:38.583 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:38.583 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:38.583 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:38.583 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:38.584 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:38.584 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:38.584 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:38.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:38.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:38.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:38.585 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:38.585 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:38.586 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:38.586 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:38.586 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:38.586 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:38.587 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:38.587 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:38.587 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:38.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:38.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:38.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:38.588 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:38.588 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:38.589 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:38.589 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:38.589 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:38.589 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:38.590 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:38.590 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:38.590 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:38.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:38.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:38.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:38.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:38.591 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:38.592 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:38.592 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:38.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:38.592 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:38.593 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:38.593 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:38.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:38.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:38.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:38.594 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:38.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:38.594 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:38.595 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:38.595 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:38.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:38.595 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:38Z" 2022-07-18T03:07:38.596 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:38.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:38Z" } 2022-07-18T03:07:38.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:38.598 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:38.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:38.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:43.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:43.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:43.602 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:43.657 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:43.657 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:43.720 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:43.721 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:43.721 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:43.721 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:43.721 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:43.722 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:43.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:43.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:43.723 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:43.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:43.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:43.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:43.724 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:43.724 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:43.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:43.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:43.725 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:43.725 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:43.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:43.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:43.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:43.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:43.726 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:43.727 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:43.727 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:43.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:43.728 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:43.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:43.728 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:43.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:43.729 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:43.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:43.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:43.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:43.730 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:43.730 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:43.730 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:43.731 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:43.731 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:43.731 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:43.731 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:43.732 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:43.732 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:43.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:43.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:43.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:43.733 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:43.733 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:43.734 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:43.734 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:43.734 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:43.734 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:43.735 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:43.735 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:43.735 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:43.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:43.736 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:43.736 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:43.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:43.736 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:43.737 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:43.737 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:43.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:43.737 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:43Z" 2022-07-18T03:07:43.738 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:43.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:43Z" } 2022-07-18T03:07:43.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:43.740 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:43.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:43.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:48.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:48.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:48.744 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:48.793 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:48.793 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:48.860 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:48.861 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:48.861 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:48.861 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:48.862 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:48.862 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:48.862 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:48.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:48.863 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:48.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:48.863 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:48.864 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:48.864 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:48.864 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:48.864 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:48.865 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:48.865 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:48.865 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:48.865 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:48.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:48.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:48.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:48.866 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:48.867 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:48.867 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:48.867 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:48.868 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:48.868 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:48.868 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:48.869 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:48.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:48.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:48.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:48.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:48.870 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:48.870 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:48.871 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:48.871 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:48.871 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:48.871 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:48.872 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:48.872 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:48.872 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:48.872 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:48.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:48.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:48.873 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:48.873 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:48.874 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:48.874 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:48.874 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:48.874 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:48.875 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:48.875 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:48.875 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:48.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:48.876 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:48.876 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:48.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:48.877 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:48.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:48.877 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:48.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:48.878 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:48Z" 2022-07-18T03:07:48.878 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:48.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:48Z" } 2022-07-18T03:07:48.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:48.881 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:48.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:48.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:53.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:53.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:53.884 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:53.959 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:53.959 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:54.020 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:54.021 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:54.021 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:54.021 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:54.021 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:54.022 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:54.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:54.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:54.023 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:54.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:54.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:54.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:54.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:54.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:54.025 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:54.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:54.025 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:54.025 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:54.026 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:54.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:54.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:54.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:54.027 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:54.027 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:54.027 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:54.027 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:54.028 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:54.028 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:54.028 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:54.028 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:54.029 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:54.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:54.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:54.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:54.030 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:54.030 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:54.030 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:54.030 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:54.031 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:54.031 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:54.031 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:54.032 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:54.032 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:54.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:54.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:54.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:54.033 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:54.033 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:54.034 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:54.034 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:54.034 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:54.034 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:54.035 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:54.035 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:54.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:54.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:54.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:54.036 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:54.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:54.037 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:54.037 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:54.037 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:54.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:54.038 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:54Z" 2022-07-18T03:07:54.038 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:54.039 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:54Z" } 2022-07-18T03:07:54.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:54.040 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:54.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:54.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:07:59.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:59.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:07:59.045 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:07:59.102 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:07:59.102 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:07:59.224 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:07:59.225 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:07:59.225 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:07:59.225 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:07:59.226 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:07:59.226 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:07:59.226 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:59.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:59.227 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:07:59.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:59.228 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:07:59.228 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:59.228 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:59.228 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:07:59.229 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:59.229 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:07:59.229 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:59.230 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:07:59.230 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:59.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:59.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:59.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:59.231 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:07:59.231 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:59.231 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:59.232 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:59.232 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:59.232 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:59.232 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:59.233 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:59.233 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:59.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:59.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:59.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:59.234 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:07:59.234 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:59.234 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:59.235 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:07:59.235 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:59.235 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:59.235 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:59.236 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:07:59.236 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:59.236 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:59.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:59.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:07:59.237 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:07:59.237 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:07:59.238 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:07:59.238 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:07:59.238 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:07:59.239 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:07:59.239 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:07:59.239 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:07:59.239 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:07:59.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:59.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:07:59.240 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:07:59.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:59.241 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:07:59.241 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:07:59.241 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:07:59.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:07:59.242 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:07:59Z" 2022-07-18T03:07:59.242 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:07:59.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:07:59Z" } 2022-07-18T03:07:59.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:07:59.245 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:07:59.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:07:59.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:04.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:04.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:04.249 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:04.304 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:04.305 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:04.380 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:04.380 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:04.380 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:04.381 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:04.381 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:04.381 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:04.381 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:04.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:04.382 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:04.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:04.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:04.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:04.383 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:04.383 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:04.383 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:04.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:04.384 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:04.384 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:04.384 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:04.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:04.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:04.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:04.385 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:04.386 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:04.386 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:04.386 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:04.387 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:04.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:04.387 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:04.387 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:04.388 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:04.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:04.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:04.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:04.389 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:04.389 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:04.389 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:04.389 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:04.390 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:04.390 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:04.390 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:04.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:04.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:04.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:04.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:04.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:04.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:04.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:04.392 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:04.392 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:04.392 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:04.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:04.393 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:04.393 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:04.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:04.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:04.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:04.394 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:04.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:04.395 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:04.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:04.395 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:04.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:04.396 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:04Z" 2022-07-18T03:08:04.396 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:04.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:04Z" } 2022-07-18T03:08:04.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:04.399 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:04.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:04.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:09.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:09.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:09.402 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:09.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:09.451 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:09.537 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:09.538 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:09.538 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:09.538 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:09.539 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:09.539 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:09.539 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:09.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:09.540 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:09.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:09.540 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:09.542 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:09.542 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:09.542 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:09.544 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:09.544 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:09.544 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:09.545 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:09.545 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:09.545 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:09.545 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:09.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:09.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:09.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:09.547 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:09.547 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:09.547 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:09.547 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:09.548 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:09.548 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:09.548 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:09.548 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:09.549 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:09.549 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:09.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:09.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:09.550 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:09.550 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:09.550 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:09.550 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:09.551 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:09.551 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:09.551 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:09.552 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:09.552 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:09.552 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:09.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:09.553 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:09.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:09.553 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:09Z" 2022-07-18T03:08:09.555 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:09.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:09Z" } 2022-07-18T03:08:09.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:09.557 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:09.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:09.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:14.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:14.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:14.562 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:14.634 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:14.635 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:14.708 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:14.709 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:14.709 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:14.710 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:14.710 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:14.711 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:14.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:14.712 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:14.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:14.713 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:14.713 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:14.714 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:14.714 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:14.715 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:14.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:14.716 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:14.717 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:14.717 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:14.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:14.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:14.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:14.719 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:14.720 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:14.720 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:14.721 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:14.721 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:14.722 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:14.722 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:14.723 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:14.723 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:14.724 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:14.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:14.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:14.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:14.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:14.727 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:14.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:14.728 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:14.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:14.729 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:14.729 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:14.730 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:14.730 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:14.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:14.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:14.732 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:14.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:14.733 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:14.733 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:14.734 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:14.734 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:14.735 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:14.735 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:14.736 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:14.736 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:14.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:14.737 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:14.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:14.738 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:14.739 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:14.739 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:14.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:14.740 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:14Z" 2022-07-18T03:08:14.741 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:14.743 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:14Z" } 2022-07-18T03:08:14.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:14.746 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:14.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:14.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:19.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:19.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:19.751 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:19.802 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:19.802 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:19.867 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:19.867 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:19.867 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:19.868 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:19.868 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:19.868 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:19.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:19.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:19.869 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:19.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:19.870 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:19.870 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:19.870 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:19.870 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:19.871 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:19.871 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:19.871 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:19.871 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:19.872 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:19.872 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:19.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:19.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:19.873 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:19.873 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:19.873 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:19.873 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:19.874 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:19.874 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:19.874 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:19.875 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:19.875 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:19.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:19.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:19.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:19.876 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:19.876 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:19.876 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:19.878 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:19.878 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:19.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:19.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:19.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:19.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:19.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:19.879 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:19.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:19.880 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:19.880 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:19.880 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:19.881 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:19.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:19.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:19.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:19.882 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:19.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:19.882 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:19.882 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:19.883 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:19.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:19.883 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:19Z" 2022-07-18T03:08:19.883 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:19.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:19Z" } 2022-07-18T03:08:19.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:19.886 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:19.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:19.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:24.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:24.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:24.891 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:24.998 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:24.998 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:24.999 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:24.999 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:24.999 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:25.000 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:25.000 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:25.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:25.000 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:25.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:25.001 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:25.001 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:25.002 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:25.002 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:25.002 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:25.003 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:25.003 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:25.003 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:25.003 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:25.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:25.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:25.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:25.005 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:25.005 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:25.005 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:25.005 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:25.006 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:25.006 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:25.006 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:25.006 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:25.007 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:25.007 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:25.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:25.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:25.008 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:25.008 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:25.008 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:25.009 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:25.009 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:25.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:25.009 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:25.010 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:25.010 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:25.010 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:25.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:25.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:25.011 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:25.012 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:25.012 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:25.012 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:25.012 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:25.013 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:25.013 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:25.013 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:25.014 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:25.014 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:25.014 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:25.015 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:25.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:25.015 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:25.015 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:25.016 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:25.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:25.016 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:24Z" 2022-07-18T03:08:25.016 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:25.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:24Z" } 2022-07-18T03:08:25.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:25.019 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:25.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:25.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:30.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:30.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:30.023 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:30.077 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:30.077 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:30.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:30.145 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:30.145 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:30.145 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:30.146 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:30.146 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:30.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:30.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:30.147 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:30.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:30.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:30.147 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:30.148 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:30.148 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:30.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:30.148 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:30.149 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:30.149 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:30.149 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:30.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:30.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:30.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:30.150 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:30Z" 2022-07-18T03:08:30.161 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:30.162 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:30Z" } 2022-07-18T03:08:30.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:30.164 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:30.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:30.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:35.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:35.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:35.169 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:35.218 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:35.218 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:35.343 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:35.343 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:35Z" 2022-07-18T03:08:35.364 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:35.365 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:35Z" } 2022-07-18T03:08:35.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:35.367 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:35.367 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:35.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:40.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:40.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:40.371 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:40.441 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:40.442 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:40.559 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:40.560 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:40.560 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:40.561 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:40.561 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:40.561 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:40.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:40.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:40.562 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:40.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:40.563 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:40.563 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:40.563 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:40.564 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:40.564 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:40.564 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:40.564 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:40.565 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:40.565 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:40.565 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:40.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:40.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:40.566 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:40.566 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:40.567 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:40.567 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:40.567 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:40.568 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:40.568 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:40.568 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:40.568 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:40.569 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:40.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:40.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:40.569 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:40.570 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:40.570 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:40.570 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:40.571 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:40.571 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:40.571 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:40.571 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:40.572 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:40.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:40.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:40.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:40.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:40.573 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:40.573 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:40.573 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:40.574 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:40.574 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:40.574 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:40.575 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:40.575 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:40.575 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:40.575 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:40.576 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:40.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:40.576 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:40.576 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:40.577 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:40.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:40.577 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:40Z" 2022-07-18T03:08:40.578 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:40.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:40Z" } 2022-07-18T03:08:40.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:40.580 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:40.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:40.581 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:45.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:45.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:45.585 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:45.633 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:45.634 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:45.707 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:45.708 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:45.708 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:45.709 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:45.709 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:45.710 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:45.710 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:45.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:45.711 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:45.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:45.712 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:45.713 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:45.713 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:45.714 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:45.714 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:45.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:45.715 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:45.716 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:45.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:45.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:45.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:45.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:45.718 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:45.719 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:45.719 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:45.720 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:45.720 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:45.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:45.721 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:45.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:45.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:45.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:45.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:45.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:45.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:45.725 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:45.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:45.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:45.727 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:45.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:45.728 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:45.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:45.729 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:45.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:45.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:45.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:45.731 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:45.731 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:45.732 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:45.732 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:45.733 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:45.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:45.734 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:45.734 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:45.735 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:45.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:45.736 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:45.736 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:45.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:45.737 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:45.738 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:45.738 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:45.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:45.739 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:45Z" 2022-07-18T03:08:45.740 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:45.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:45Z" } 2022-07-18T03:08:45.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:45.744 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:45.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:45.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:50.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:50.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:50.750 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:50.798 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:50.799 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:50.868 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:50.868 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:50.868 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:50.869 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:50.869 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:50.869 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:50.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:50.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:50.870 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:50.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:50.871 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:50.871 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:50.871 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:50.871 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:50.872 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:50.872 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:50.872 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:50.872 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:50.873 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:50.873 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:50.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:50.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:50.874 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:50.874 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:50.875 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:50.875 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:50.876 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:50.876 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:50.877 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:50.877 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:50.878 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:50.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:50.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:50.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:50.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:50.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:50.879 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:50.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:50.880 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:50.880 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:50.880 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:50.881 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:50.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:50.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:50.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:50.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:50.882 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:50.882 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:50.882 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:50.883 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:50.883 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:50.883 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:50.884 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:50.884 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:50.885 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:50.885 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:50.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:50.888 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:50.888 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:50.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:50.889 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:50Z" 2022-07-18T03:08:50.889 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:50.891 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:50Z" } 2022-07-18T03:08:50.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:50.892 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:50.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:50.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:08:55.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:55.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:08:55.897 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:08:55.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:08:55.948 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:08:56.007 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:08:56.008 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:08:56.008 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:08:56.008 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:08:56.008 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:08:56.009 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:08:56.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:56.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:56.010 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:08:56.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:56.010 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:08:56.011 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:56.011 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:56.011 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:08:56.011 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:56.012 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:08:56.012 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:56.012 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:08:56.012 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:56.013 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:56.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:56.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:56.014 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:08:56.014 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:56.014 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:56.014 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:56.015 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:56.015 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:56.015 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:56.015 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:56.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:56.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:56.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:56.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:56.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:08:56.017 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:56.017 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:56.017 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:08:56.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:56.018 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:56.018 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:56.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:08:56.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:56.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:56.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:56.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:08:56.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:08:56.021 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:08:56.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:08:56.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:08:56.022 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:08:56.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:08:56.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:08:56.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:08:56.023 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:08:56.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:56.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:08:56.024 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:08:56.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:56.024 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:08:56.024 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:08:56.025 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:08:56.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:08:56.025 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:08:55Z" 2022-07-18T03:08:56.025 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:08:56.027 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:08:55Z" } 2022-07-18T03:08:56.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:08:56.028 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:08:56.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:08:56.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:01.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:01.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:01.031 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:01.083 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:01.083 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:01.139 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:01.140 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:01.140 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:01.140 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:01.141 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:01.141 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:01.141 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:01.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:01.142 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:01.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:01.142 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:01.143 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:01.143 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:01.143 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:01.143 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:01.144 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:01.144 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:01.144 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:01.145 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:01.145 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:01.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:01.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:01.146 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:01.146 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:01.146 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:01.147 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:01.147 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:01.147 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:01.147 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:01.148 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:01.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:01.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:01.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:01.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:01.149 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:01.149 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:01.150 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:01.150 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:01.150 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:01.150 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:01.151 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:01.151 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:01.151 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:01.151 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:01.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:01.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:01.152 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:01.152 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:01.153 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:01.153 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:01.153 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:01.154 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:01.154 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:01.154 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:01.154 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:01.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:01.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:01.155 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:01.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:01.156 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:01.156 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:01.156 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:01Z" 2022-07-18T03:09:01.157 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:01.158 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:01Z" } 2022-07-18T03:09:01.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:01.160 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:01.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:01.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:06.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:06.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:06.163 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:06.235 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:06.318 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:06.319 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:06.319 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:06.319 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:06.319 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:06.320 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:06.320 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:06.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:06.320 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:06.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:06.321 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:06.321 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:06.321 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:06.322 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:06.322 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:06.322 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:06.322 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:06.323 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:06.323 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:06.323 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:06.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:06.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:06.324 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:06.324 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:06.324 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:06.325 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:06.325 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:06.325 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:06.325 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:06.326 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:06.326 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:06.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:06.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:06.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:06.327 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:06.327 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:06.327 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:06.328 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:06.328 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:06.328 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:06.328 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:06.329 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:06.329 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:06.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:06.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:06.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:06.330 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:06.330 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:06.330 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:06.331 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:06.331 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:06.331 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:06.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:06.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:06.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:06.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:06.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:06.333 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:06.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:06.333 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:06.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:06.334 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:06.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:06.334 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:06Z" 2022-07-18T03:09:06.334 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:06.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:06Z" } 2022-07-18T03:09:06.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:06.337 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:06.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:06.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:11.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:11.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:11.341 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:11.391 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:11.391 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:11.469 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:11.469 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:11.469 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:11.470 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:11.470 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:11.470 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:11.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:11.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:11.471 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:11.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:11.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:11.472 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:11.472 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:11.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:11.473 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:11.473 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:11.474 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:11.474 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:11.474 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:11.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:11.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:11.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:11.475 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:11.476 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:11.476 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:11.476 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:11.476 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:11.477 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:11.477 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:11.477 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:11.478 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:11.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:11.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:11.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:11.479 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:11.479 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:11.479 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:11.480 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:11.480 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:11.480 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:11.481 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:11.481 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:11.481 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:11.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:11.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:11.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:11.482 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:11.483 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:11.483 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:11.483 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:11.483 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:11.484 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:11.484 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:11.484 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:11.484 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:11.485 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:11.485 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:11.485 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:11.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:11.486 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:11.486 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:11.486 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:11.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:11.487 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:11Z" 2022-07-18T03:09:11.487 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:11.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:11Z" } 2022-07-18T03:09:11.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:11.489 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:11.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:11.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:16.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:16.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:16.493 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:16.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:16.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:16.606 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:16.611 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:16.611 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:16.611 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:16.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:16.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:16.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:16.612 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:16.612 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:16.613 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:16.613 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:16.613 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:16.613 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:16.614 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:16.614 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:16.614 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:16.614 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:16.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:16.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:16.615 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:16.615 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:16.616 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:16.616 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:16.616 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:16.616 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:16.617 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:16.617 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:16.617 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:16.617 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:16.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:16.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:16.618 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:16.618 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:16.619 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:16.619 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:16.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:16.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:16.620 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:16.620 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:16.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:16.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:16.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:16.622 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:16.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:16.622 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:16.623 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:16.623 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:16.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:16.624 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:16Z" 2022-07-18T03:09:16.624 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:16.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:16Z" } 2022-07-18T03:09:16.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:16.627 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:16.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:16.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:21.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:21.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:21.631 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:21.703 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:21.704 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:21.788 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:21.788 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:21.788 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:21.789 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:21.789 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:21.789 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:21.790 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:21.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:21.790 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:21.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:21.791 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:21.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:21.791 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:21.791 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:21.792 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:21.792 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:21.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:21.792 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:21.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:21.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:21.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:21.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:21.794 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:21.794 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:21.794 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:21.794 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:21.795 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:21.795 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:21.795 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:21.795 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:21.796 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:21.796 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:21.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:21.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:21.797 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:21.797 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:21.797 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:21.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:21.798 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:21.798 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:21.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:21.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:21.799 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:21.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:21.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:21.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:21.800 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:21.800 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:21.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:21.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:21.801 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:21.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:21.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:21.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:21.802 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:21.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:21.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:21.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:21.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:21.803 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:21.803 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:21.804 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:21.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:21.804 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:21Z" 2022-07-18T03:09:21.804 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:21.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:21Z" } 2022-07-18T03:09:21.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:21.807 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:21.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:21.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:26.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:26.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:26.812 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:26.862 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:26.863 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:26.939 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:26.940 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:26.940 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:26.941 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:26.941 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:26.942 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:26.942 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:26.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:26.943 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:26.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:26.944 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:26.945 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:26.945 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:26.946 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:26.946 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:26.947 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:26.948 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:26.948 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:26.949 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:26.949 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:26.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:26.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:26.950 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:26.951 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:26.951 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:26.951 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:26.951 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:26.952 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:26.952 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:26.953 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:26.953 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:26.953 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:26.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:26.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:26.955 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:26.955 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:26.955 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:26.956 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:26.956 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:26.957 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:26.957 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:26.958 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:26.958 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:26.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:26.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:26.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:26.960 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:26.961 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:26.961 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:26.961 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:26.962 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:26.962 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:26.962 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:26.963 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:26.963 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:26.963 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:26.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:26.964 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:26.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:26.965 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:26.965 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:26.965 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:26.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:26.966 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:26Z" 2022-07-18T03:09:26.966 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:26.968 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:26Z" } 2022-07-18T03:09:26.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:26.969 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:26.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:26.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:31.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:31.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:31.974 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:32.048 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:32.048 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:32.162 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:32Z" 2022-07-18T03:09:32.180 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:32.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:32Z" } 2022-07-18T03:09:32.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:32.183 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:32.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:32.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:37.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:37.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:37.186 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:37.234 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:37.235 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:37.341 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:37.342 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:37.342 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:37.342 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:37.343 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:37.343 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:37.343 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:37.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:37.344 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:37.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:37.344 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:37.345 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:37.345 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:37.345 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:37.346 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:37.346 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:37.346 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:37.347 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:37.347 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:37.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:37.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:37.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:37.348 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:37.348 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:37.348 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:37.349 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:37.349 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:37.349 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:37.350 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:37.350 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:37.350 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:37.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:37.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:37.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:37.351 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:37.351 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:37.352 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:37.352 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:37.352 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:37.353 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:37.353 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:37.353 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:37.353 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:37.354 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:37.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:37.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:37.354 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:37.355 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:37.355 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:37.355 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:37.355 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:37.356 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:37.356 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:37.356 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:37.357 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:37.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:37.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:37.357 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:37.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:37.358 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:37.358 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:37.359 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:37.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:37.359 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:37Z" 2022-07-18T03:09:37.359 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:37.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:37Z" } 2022-07-18T03:09:37.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:37.362 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:37.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:37.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:42.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:42.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:42.366 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:42.420 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:42.420 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:42.574 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:42.575 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:42.575 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:42.576 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:42.576 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:42.576 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:42.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:42.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:42.577 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:42.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:42.577 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:42.578 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:42.578 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:42.578 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:42.579 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:42.579 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:42.579 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:42.579 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:42.580 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:42.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:42.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:42.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:42.581 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:42.581 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:42.581 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:42.582 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:42.582 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:42.582 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:42.582 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:42.583 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:42.583 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:42.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:42.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:42.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:42.584 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:42.584 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:42.584 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:42.585 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:42.585 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:42.585 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:42.585 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:42.586 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:42.586 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:42.586 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:42.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:42.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:42.587 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:42.587 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:42.587 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:42.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:42.588 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:42.588 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:42.588 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:42.589 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:42.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:42.589 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:42.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:42.590 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:42.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:42.590 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:42.591 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:42.591 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:42.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:42.591 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:42Z" 2022-07-18T03:09:42.592 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:42.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:42Z" } 2022-07-18T03:09:42.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:42.594 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:42.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:42.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:47.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:47.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:47.598 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:47.646 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:47.647 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:47.704 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:47.707 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:47.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:47.707 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:47.708 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:47.708 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:47.708 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:47.708 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:47.709 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:47.709 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:47.709 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:47.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:47.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:47.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:47.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:47.710 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:47.711 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:47.711 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:47.711 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:47.711 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:47.712 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:47.712 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:47.712 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:47.712 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:47.713 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:47.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:47.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:47.713 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:47.714 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:47.714 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:47.714 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:47.715 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:47.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:47.715 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:47.715 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:47.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:47.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:47.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:47.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:47.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:47.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:47.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:47.717 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:47.718 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:47.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:47.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:47.718 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:47.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:47.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:47.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:47.719 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:47.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:47.720 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:47.720 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:47.720 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:47.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:47.721 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:47Z" 2022-07-18T03:09:47.721 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:47.722 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:47Z" } 2022-07-18T03:09:47.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:47.724 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:47.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:47.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:52.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:52.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:52.728 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:52.783 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:52.784 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:52.840 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:52.840 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:52.840 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:52.841 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:52.841 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:52.841 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:52.842 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:52.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:52.842 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:52.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:52.843 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:52.843 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:52.843 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:52.843 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:52.844 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:52.844 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:52.844 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:52.844 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:52.845 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:52.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:52.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:52.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:52.846 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:52.846 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:52.846 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:52.846 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:52.847 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:52.847 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:52.847 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:52.847 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:52.848 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:52.848 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:52.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:52.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:52.849 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:52.849 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:52.849 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:52.849 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:52.850 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:52.850 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:52.850 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:52.850 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:52.851 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:52.851 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:52.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:52.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:52.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:52.852 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:52.852 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:52.852 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:52.853 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:52.853 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:52.853 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:52.853 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:52.854 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:52.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:52.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:52.854 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:52.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:52.855 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:52.855 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:52.855 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:52.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:52.856 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:52Z" 2022-07-18T03:09:52.856 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:52.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:52Z" } 2022-07-18T03:09:52.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:52.859 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:52.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:52.859 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:09:57.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:57.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:09:57.862 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:09:57.920 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:09:57.920 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:09:57.997 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:09:57.998 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:09:57.998 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:09:57.998 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:09:57.999 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:09:57.999 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:09:57.999 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:57.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:58.000 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:09:58.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:58.000 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:09:58.000 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:58.001 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:58.001 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:09:58.001 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:58.001 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:09:58.002 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:58.002 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:09:58.002 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:58.003 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:58.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:58.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:58.003 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:09:58.004 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:58.004 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:58.004 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:58.004 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:58.005 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:58.005 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:58.005 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:58.005 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:58.006 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:58.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:58.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:58.006 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:09:58.007 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:58.007 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:58.007 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:09:58.007 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:58.008 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:58.008 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:58.008 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:09:58.008 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:58.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:58.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:58.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:09:58.009 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:09:58.010 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:09:58.010 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:09:58.010 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:09:58.010 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:09:58.011 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:09:58.011 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:09:58.011 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:09:58.011 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:09:58.012 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:58.012 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:09:58.012 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:09:58.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:58.013 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:09:58.013 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:09:58.013 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:09:58.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:09:58.014 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:09:57Z" 2022-07-18T03:09:58.014 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:09:58.015 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:09:57Z" } 2022-07-18T03:09:58.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:09:58.016 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:09:58.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:09:58.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:03.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:03.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:03.020 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:03.102 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:03.102 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:03.287 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:03.288 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:03.288 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:03.288 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:03.289 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:03.289 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:03.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:03.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:03.290 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:03.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:03.291 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:03.291 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:03.292 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:03.292 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:03.292 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:03.293 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:03.293 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:03.294 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:03.294 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:03.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:03.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:03.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:03.295 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:03.296 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:03.296 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:03.297 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:03.297 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:03.297 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:03.298 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:03.298 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:03.298 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:03.299 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:03.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:03.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:03.300 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:03.300 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:03.301 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:03.301 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:03.301 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:03.302 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:03.302 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:03.302 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:03.303 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:03.303 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:03.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:03.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:03.304 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:03.305 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:03.305 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:03.305 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:03.306 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:03.306 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:03.306 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:03.307 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:03.307 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:03.307 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:03.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:03.308 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:03.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:03.309 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:03.309 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:03.310 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:03.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:03.311 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:03Z" 2022-07-18T03:10:03.311 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:03.312 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:03Z" } 2022-07-18T03:10:03.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:03.314 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:03.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:03.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:08.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:08.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:08.317 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:08.366 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:08.367 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:08.439 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:08.439 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:08.440 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:08.440 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:08.440 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:08.441 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:08.441 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:08.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:08.443 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:08.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:08.444 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:08.444 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:08.445 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:08.446 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:08.446 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:08.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:08.447 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:08.448 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:08.449 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:08.449 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:08.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:08.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:08.451 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:08.451 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:08.452 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:08.453 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:08.453 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:08.454 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:08.455 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:08.455 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:08.456 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:08.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:08.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:08.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:08.458 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:08.459 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:08.459 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:08.460 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:08.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:08.461 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:08.462 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:08.462 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:08.463 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:08.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:08.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:08.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:08.465 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:08.466 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:08.467 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:08.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:08.468 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:08.468 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:08.469 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:08.469 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:08.470 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:08.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:08.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:08.472 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:08.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:08.473 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:08.474 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:08.474 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:08.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:08.476 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:08Z" 2022-07-18T03:10:08.476 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:08.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:08Z" } 2022-07-18T03:10:08.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:08.480 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:08.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:08.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:13.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:13.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:13.484 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:13.550 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:13.551 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:13.619 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:13.620 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:13.620 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:13.620 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:13.620 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:13.621 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:13.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:13.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:13.621 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:13.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:13.622 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:13.623 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:13.623 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:13.623 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:13.624 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:13.624 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:13.624 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:13.625 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:13.625 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:13.625 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:13.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:13.633 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:13.633 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:13.634 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:13.634 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:13Z" 2022-07-18T03:10:13.638 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:13.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:13Z" } 2022-07-18T03:10:13.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:13.640 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:13.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:13.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:18.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:18.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:18.644 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:18.695 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:18.696 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:18.836 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:18.836 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:18.837 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:18.837 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:18.837 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:18.838 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:18.838 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:18.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:18.838 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:18.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:18.839 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:18.839 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:18.839 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:18.840 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:18.840 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:18.840 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:18.840 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:18.841 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:18.841 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:18.841 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:18Z" 2022-07-18T03:10:18.853 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:18.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:18Z" } 2022-07-18T03:10:18.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:18.855 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:18.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:18.856 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:23.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:23.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:23.859 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:23.917 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:23.918 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:23.992 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:23.992 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:24.002 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:24.002 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:24.002 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:24.003 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:24.003 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:24.003 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:24.003 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:24.004 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:24.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:24.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:24.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:24.005 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:24.005 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:24.005 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:24.006 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:24.006 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:24.006 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:24.007 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:24.007 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:24.007 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:24.007 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:24.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:24.008 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:24.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:24.009 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:24.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:24.009 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:24.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:24.010 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:23Z" 2022-07-18T03:10:24.010 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:24.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:23Z" } 2022-07-18T03:10:24.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:24.012 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:24.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:24.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:29.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:29.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:29.018 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:29.067 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:29.067 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:29.145 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:29.145 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:29.145 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:29.146 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:29.146 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:29.146 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:29.147 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:29.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:29.147 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:29.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:29.148 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:29.148 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:29.149 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:29.149 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:29.150 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:29.150 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:29.150 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:29.150 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:29.151 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:29.151 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:29.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:29.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:29.152 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:29.152 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:29.153 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:29.153 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:29.153 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:29Z" 2022-07-18T03:10:29.164 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:29.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:29Z" } 2022-07-18T03:10:29.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:29.167 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:29.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:29.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:34.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:34.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:34.172 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:34.221 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:34.222 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:34.286 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:34.289 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:34.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:34.290 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:34.291 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:34.291 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:34.292 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:34.292 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:34.293 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:34.293 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:34.294 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:34.295 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:34.295 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:34.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:34.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:34.297 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:34.298 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:34.298 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:34.299 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:34.299 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:34.300 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:34.300 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:34.301 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:34.302 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:34.302 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:34.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:34.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:34.304 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:34.304 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:34.305 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:34.305 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:34.306 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:34.306 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:34.307 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:34.308 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:34.308 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:34.309 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:34.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:34.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:34.310 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:34.311 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:34.311 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:34.312 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:34.312 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:34.313 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:34.313 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:34.314 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:34.314 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:34.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:34.316 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:34.316 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:34.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:34.317 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:34.318 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:34.318 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:34.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:34.319 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:34Z" 2022-07-18T03:10:34.320 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:34.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:34Z" } 2022-07-18T03:10:34.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:34.324 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:34.325 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:34.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:39.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:39.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:39.330 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:39.385 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:39.386 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:39.513 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:39.513 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:39.514 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:39.514 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:39.514 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:39.514 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:39.515 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:39.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:39.515 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:39.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:39.516 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:39.516 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:39.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:39.517 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:39.517 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:39.517 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:39.517 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:39.518 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:39.518 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:39.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:39.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:39.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:39.519 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:39.519 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:39.519 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:39.520 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:39.520 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:39.520 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:39.521 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:39.521 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:39.521 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:39.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:39.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:39.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:39.522 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:39.522 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:39.523 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:39.523 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:39.523 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:39.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:39.524 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:39.524 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:39.524 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:39.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:39.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:39.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:39.525 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:39.526 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:39.526 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:39.526 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:39.526 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:39.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:39.527 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:39.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:39.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:39.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:39.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:39.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:39.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:39.529 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:39.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:39.529 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:39.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:39.529 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:39Z" 2022-07-18T03:10:39.530 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:39.531 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:39Z" } 2022-07-18T03:10:39.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:39.532 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:39.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:39.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:44.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:44.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:44.537 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:44.593 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:44.594 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:44.661 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:44.661 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:44.662 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:44.662 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:44.662 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:44.662 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:44.663 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:44.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:44.663 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:44.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:44.664 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:44.664 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:44.664 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:44.666 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:44.666 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:44.666 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:44.667 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:44.667 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:44.668 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:44.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:44.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:44.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:44.669 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:44.669 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:44.670 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:44.670 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:44.670 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:44.670 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:44.671 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:44.671 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:44.671 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:44.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:44.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:44.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:44.672 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:44.672 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:44.673 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:44.673 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:44.673 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:44.673 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:44.674 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:44.674 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:44.674 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:44.675 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:44.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:44.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:44.676 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:44.676 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:44.676 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:44.676 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:44.677 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:44.677 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:44.677 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:44.677 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:44.678 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:44.678 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:44.678 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:44.678 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:44.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:44.679 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:44.679 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:44.679 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:44.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:44.680 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:44Z" 2022-07-18T03:10:44.680 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:44.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:44Z" } 2022-07-18T03:10:44.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:44.683 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:44.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:44.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:49.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:49.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:49.687 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:49.743 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:49.743 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:49.806 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:49.807 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:49.807 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:49.808 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:49.808 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:49.809 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:49.809 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:49.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:49.810 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:49.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:49.811 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:49.811 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:49.811 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:49.812 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:49.812 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:49.812 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:49.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:49.813 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:49.813 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:49.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:49.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:49.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:49.814 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:49.815 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:49.815 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:49.815 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:49.816 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:49.816 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:49.816 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:49.816 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:49.817 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:49.817 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:49.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:49.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:49.818 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:49.818 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:49.818 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:49.819 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:49.819 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:49.820 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:49.820 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:49.821 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:49.821 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:49.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:49.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:49.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:49.824 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:49.825 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:49.825 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:49.826 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:49.826 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:49.827 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:49.827 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:49.828 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:49.828 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:49.829 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:49.829 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:49.830 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:49.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:49.831 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:49.831 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:49.832 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:49.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:49.833 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:49Z" 2022-07-18T03:10:49.833 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:49.835 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:49Z" } 2022-07-18T03:10:49.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:49.837 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:49.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:49.839 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:54.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:54.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:54.842 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:10:54.899 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:10:54.900 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:10:54.963 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:10:54.963 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:10:54.964 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:10:54.964 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:10:54.964 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:10:54.965 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:10:54.965 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:54.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:10:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:10:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:10:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:10:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:10:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:10:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:10:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:10:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:10:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:10:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:10:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:10:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:10:54.977 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:10:54.977 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:10:54.977 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:10:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:10:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:10:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:10:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:10:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:10:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:10:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:10:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:10:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:10:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:10:54Z" 2022-07-18T03:10:54.981 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:10:54.982 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:10:54Z" } 2022-07-18T03:10:54.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:10:54.983 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:10:54.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:10:54.984 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:10:59.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:59.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:10:59.987 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:00.044 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:00.045 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:00.113 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:00.114 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:00.114 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:00.115 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:00.115 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:00.116 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:00.116 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:00.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:00.116 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:00.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:00.118 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:00.119 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:00.119 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:00.120 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:00.120 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:00.120 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:00.121 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:00.121 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:00.122 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:00.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:00.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:00.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:00.123 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:00.123 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:00.123 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:00.123 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:00.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:00.124 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:00.124 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:00.124 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:00.124 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:00.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:00.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:00.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:00.126 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:00.127 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:00.127 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:00.128 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:00.128 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:00.129 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:00.129 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:00.130 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:00.130 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:00.131 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:00.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:00.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:00.132 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:00.133 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:00.133 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:00.134 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:00.134 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:00.135 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:00.135 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:00.136 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:00.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:00.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:00.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:00.138 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:00.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:00.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:00.140 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:00Z" 2022-07-18T03:11:00.141 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:00.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:00Z" } 2022-07-18T03:11:00.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:00.146 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:00.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:00.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:05.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:05.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:05.151 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:05.202 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:05.203 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:05.273 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:05.274 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:05.274 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:05.275 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:05.275 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:05.275 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:05.275 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:05.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:05.276 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:05.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:05.276 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:05.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:05.287 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:05.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:05.290 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:05.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:05.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:05.292 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:05Z" 2022-07-18T03:11:05.292 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:05.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:05Z" } 2022-07-18T03:11:05.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:05.297 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:05.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:05.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:10.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:10.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:10.303 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:10.351 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:10.352 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:10.412 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:10.412 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:10.412 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:10.413 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:10.413 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:10.413 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:10.414 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:10.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:10.414 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:10.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:10.415 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:10.415 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:10.415 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:10.416 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:10.416 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:10.416 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:10.417 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:10.417 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:10.417 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:10.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:10.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:10.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:10.419 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:10.419 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:10.420 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:10.420 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:10.420 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:10.420 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:10.421 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:10.421 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:10.422 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:10.423 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:10.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:10.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:10.424 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:10.424 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:10.425 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:10.425 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:10.426 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:10.426 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:10.427 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:10.427 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:10.428 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:10.428 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:10.430 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:10.430 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:10.431 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:10.431 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:10.433 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:10.433 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:10.436 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:10Z" 2022-07-18T03:11:10.436 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:10.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:10Z" } 2022-07-18T03:11:10.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:10.439 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:10.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:10.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:15.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:15.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:15.443 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:15.494 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:15.495 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:15.635 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:15.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:15.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:15.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:15.652 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:15.653 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:15.653 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:15.654 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:15.654 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:15.655 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:15.655 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:15.656 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:15.656 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:15.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:15.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:15.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:15.659 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:15.662 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:15.663 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:15.664 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:15.664 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:15.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:15.665 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:15.666 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:15.666 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:15.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:15.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:15.672 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:15.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:15.673 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:15.673 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:15.674 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:15.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:15.674 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:15Z" 2022-07-18T03:11:15.675 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:15.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:15Z" } 2022-07-18T03:11:15.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:15.678 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:15.679 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:15.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:20.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:20.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:20.682 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:20.730 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:20.731 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:20.794 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:20.806 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:20.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:20.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:20.807 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:20.808 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:20.808 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:20.809 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:20.809 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:20.810 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:20.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:20.813 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:20.813 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:20.820 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:20.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:20.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:20.821 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:20.822 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:20.822 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:20.823 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:20.823 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:20.823 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:20.824 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:20.824 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:20.825 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:20.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:20.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:20.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:20Z" 2022-07-18T03:11:20.831 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:20.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:20Z" } 2022-07-18T03:11:20.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:20.834 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:20.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:20.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:25.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:25.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:25.841 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:25.897 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:25.898 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:25.976 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:25.976 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:25.976 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:25.977 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:25.977 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:25.978 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:25.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:25.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:25.982 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:25.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:25.983 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:25.984 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:25.984 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:25.985 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:25.985 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:25.986 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:25.987 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:25.987 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:25.988 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:25.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:25.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:25.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:25.991 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:25.992 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:25.992 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:25.993 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:25.994 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:25.994 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:25.995 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:25.995 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:25.996 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:25.997 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:25.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:25.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:25.999 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:25.999 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:26.000 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:26.001 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:26.001 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:26.002 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:26.002 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:26.003 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:26.004 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:26.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:26.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:26.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:26.006 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:26.006 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:26.007 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:26.008 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:26.008 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:26.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:26.009 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:26.010 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:26.010 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:26.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:26.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:26.012 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:26.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:26.012 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:26.013 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:26.013 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:26.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:26.014 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:25Z" 2022-07-18T03:11:26.014 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:26.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:25Z" } 2022-07-18T03:11:26.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:26.017 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:26.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:26.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:31.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:31.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:31.022 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:31.080 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:31.081 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:31.155 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:31.156 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:31.157 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:31.157 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:31.158 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:31.158 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:31.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:31.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:31.160 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:31.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:31.161 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:31.161 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:31.162 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:31.162 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:31.163 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:31.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:31.163 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:31.164 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:31.164 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:31.168 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:31.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:31.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:31.170 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:31.170 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:31.170 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:31.170 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:31.171 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:31.171 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:31.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:31.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:31.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:31.172 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:31.172 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:31.174 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:31.174 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:31.175 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:31.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:31.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:31.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:31.176 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:31.177 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:31.177 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:31.178 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:31.178 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:31.179 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:31.179 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:31.179 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:31.180 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:31.180 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:31.181 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:31.181 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:31.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:31.182 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:31.183 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:31.183 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:31.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:31.184 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:31Z" 2022-07-18T03:11:31.184 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:31.185 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:31Z" } 2022-07-18T03:11:31.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:31.186 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:31.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:31.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:36.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:36.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:36.191 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:36.248 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:36.248 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:36.317 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:36.318 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:36.319 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:36.319 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:36.320 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:36.320 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:36.321 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:36.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:36.322 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:36.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:36.323 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:36.324 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:36.324 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:36.325 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:36.325 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:36.326 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:36.327 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:36.327 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:36.328 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:36.328 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:36.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:36.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:36.329 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:36.329 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:36.329 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:36.330 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:36.330 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:36.330 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:36.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:36.331 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:36.331 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:36.331 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:36.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:36.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:36.332 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:36.332 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:36.333 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:36.333 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:36.333 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:36.334 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:36.334 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:36.335 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:36.335 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:36.336 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:36.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:36.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:36.338 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:36.338 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:36.339 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:36.339 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:36.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:36.340 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:36.340 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:36.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:36.341 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:36.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:36.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:36.342 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:36.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:36.342 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:36.343 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:36.343 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:36.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:36.344 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:36Z" 2022-07-18T03:11:36.344 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:36.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:36Z" } 2022-07-18T03:11:36.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:36.348 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:36.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:36.349 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:41.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:41.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:41.353 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:41.404 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:41.404 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:41.469 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:41.470 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:41.470 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:41.471 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:41.472 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:41.472 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:41.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:41.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:41.474 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:41.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:41Z" 2022-07-18T03:11:41.496 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:41.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:41Z" } 2022-07-18T03:11:41.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:41.498 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:41.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:41.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:46.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:46.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:46.506 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:46.572 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:46.572 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:46.634 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:46Z" 2022-07-18T03:11:46.679 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:46.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:46Z" } 2022-07-18T03:11:46.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:46.681 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:46.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:46.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:51.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:51.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:51.707 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:51.770 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:51.771 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:51.847 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:51.848 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:51.849 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:51.849 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:51.850 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:51.851 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:51.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:51.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:51.853 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:51.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:51.855 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:51.855 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:51.856 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:51.856 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:51.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:51.858 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:51.858 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:51.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:51.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:51.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:51.887 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:51.887 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:51.888 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:51.888 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:51.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:51.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:51.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:51.891 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:51.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:51.892 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:51.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:51.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:51.894 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:51.894 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:51.895 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:51.895 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:51.896 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:51.896 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:51.897 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:51.897 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:51.906 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:51.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:51.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:51.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:51.909 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:51.909 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:51.910 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:51.911 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:51.911 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:51.912 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:51.912 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:51.913 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:51.913 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:51.914 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:51.915 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:51.915 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:51.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:51.916 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:51.917 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:51.917 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:51.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:51.918 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:51Z" 2022-07-18T03:11:51.919 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:51.922 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:51Z" } 2022-07-18T03:11:51.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:51.925 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:51.926 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:51.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:11:56.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... FAIL 2022-07-18T03:11:56.979 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-10 2022-07-18T03:11:56.981 DEBUG:boto:path=/mlvank-10/ 2022-07-18T03:11:56.987 DEBUG:boto:auth_path=/mlvank-10/ 2022-07-18T03:11:56.994 DEBUG:boto:Method: PUT 2022-07-18T03:11:56.995 DEBUG:boto:Path: /mlvank-10/ 2022-07-18T03:11:56.995 DEBUG:boto:Data: 2022-07-18T03:11:56.996 DEBUG:boto:Headers: {} 2022-07-18T03:11:56.997 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:56.997 DEBUG:boto:Port: 8000 2022-07-18T03:11:56.997 DEBUG:boto:Params: {} 2022-07-18T03:11:57.000 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:11:57.000 DEBUG:boto:Token: None 2022-07-18T03:11:57.001 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/ 2022-07-18T03:11:57.002 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:fInFN7K8Mfxd0ABev1xtxdbwWwg= 2022-07-18T03:11:57.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:fInFN7K8Mfxd0ABev1xtxdbwWwg=', 'Content-Length': '0'} 2022-07-18T03:11:57.030 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000de517c33f61f744-0062d4cf7d-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.031 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-11 2022-07-18T03:11:57.032 DEBUG:boto:path=/mlvank-11/ 2022-07-18T03:11:57.032 DEBUG:boto:auth_path=/mlvank-11/ 2022-07-18T03:11:57.033 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.033 DEBUG:boto:Path: /mlvank-11/ 2022-07-18T03:11:57.035 DEBUG:boto:Data: 2022-07-18T03:11:57.036 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.037 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:57.037 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.038 DEBUG:boto:Params: {} 2022-07-18T03:11:57.038 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:11:57.039 DEBUG:boto:Token: None 2022-07-18T03:11:57.040 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:11:57 GMT /mlvank-11/ 2022-07-18T03:11:57.040 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:z60accVCzUcFAgvwfZijdmbr/8M= 2022-07-18T03:11:57.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:z60accVCzUcFAgvwfZijdmbr/8M=', 'Content-Length': '0'} 2022-07-18T03:11:57.105 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d4a5b17841cc194-0062d4cf7d-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.106 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-12 2022-07-18T03:11:57.108 DEBUG:boto:path=/mlvank-12/ 2022-07-18T03:11:57.109 DEBUG:boto:auth_path=/mlvank-12/ 2022-07-18T03:11:57.110 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.111 DEBUG:boto:Path: /mlvank-12/ 2022-07-18T03:11:57.112 DEBUG:boto:Data: 2022-07-18T03:11:57.113 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.120 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:57.121 DEBUG:boto:Port: 8001 2022-07-18T03:11:57.122 DEBUG:boto:Params: {} 2022-07-18T03:11:57.124 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:11:57.125 DEBUG:boto:Token: None 2022-07-18T03:11:57.135 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:11:57 GMT /mlvank-12/ 2022-07-18T03:11:57.136 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:WJy6XtDKHqtgtrP7vEVdqKIvaV8= 2022-07-18T03:11:57.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:WJy6XtDKHqtgtrP7vEVdqKIvaV8=', 'Content-Length': '0'} 2022-07-18T03:11:57.182 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000530e140836537fbf-0062d4cf7d-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.184 DEBUG:boto:path=/mlvank-10/ 2022-07-18T03:11:57.184 DEBUG:boto:auth_path=/mlvank-10/ 2022-07-18T03:11:57.185 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.186 DEBUG:boto:Path: /mlvank-10/ 2022-07-18T03:11:57.187 DEBUG:boto:Data: 2022-07-18T03:11:57.188 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.189 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.194 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.196 DEBUG:boto:Params: {} 2022-07-18T03:11:57.197 DEBUG:boto:Token: None 2022-07-18T03:11:57.198 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/ 2022-07-18T03:11:57.199 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8= 2022-07-18T03:11:57.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8=', 'Content-Length': '0'} 2022-07-18T03:11:57.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063209a8dd684c352-0062d4cf7d-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.232 DEBUG:boto:path=/mlvank-10/myobj 2022-07-18T03:11:57.245 DEBUG:boto:auth_path=/mlvank-10/myobj 2022-07-18T03:11:57.246 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.247 DEBUG:boto:Path: /mlvank-10/myobj 2022-07-18T03:11:57.248 DEBUG:boto:Data: 2022-07-18T03:11:57.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:57.253 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.263 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.264 DEBUG:boto:Params: {} 2022-07-18T03:11:57.265 DEBUG:boto:Token: None 2022-07-18T03:11:57.266 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/myobj 2022-07-18T03:11:57.267 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:6Ck6fc1NM96MvvIwQ5br0NqUU7c= 2022-07-18T03:11:57.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:6Ck6fc1NM96MvvIwQ5br0NqUU7c='} 2022-07-18T03:11:57.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e49a72111d0e550-0062d4cf7d-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.301 DEBUG:boto:path=/mlvank-10/ 2022-07-18T03:11:57.302 DEBUG:boto:auth_path=/mlvank-10/ 2022-07-18T03:11:57.303 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.304 DEBUG:boto:Path: /mlvank-10/ 2022-07-18T03:11:57.305 DEBUG:boto:Data: 2022-07-18T03:11:57.306 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.307 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.308 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.309 DEBUG:boto:Params: {} 2022-07-18T03:11:57.310 DEBUG:boto:Token: None 2022-07-18T03:11:57.311 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/ 2022-07-18T03:11:57.312 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8= 2022-07-18T03:11:57.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8=', 'Content-Length': '0'} 2022-07-18T03:11:57.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffa8c048c130c87a-0062d4cf7d-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.337 DEBUG:boto:path=/mlvank-10/_myobj 2022-07-18T03:11:57.340 DEBUG:boto:auth_path=/mlvank-10/_myobj 2022-07-18T03:11:57.341 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.356 DEBUG:boto:Path: /mlvank-10/_myobj 2022-07-18T03:11:57.357 DEBUG:boto:Data: 2022-07-18T03:11:57.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:57.359 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.360 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.361 DEBUG:boto:Params: {} 2022-07-18T03:11:57.389 DEBUG:boto:Token: None 2022-07-18T03:11:57.390 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/_myobj 2022-07-18T03:11:57.391 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:weyid9XEasTcO+mabMVEmwmDSy4= 2022-07-18T03:11:57.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:weyid9XEasTcO+mabMVEmwmDSy4='} 2022-07-18T03:11:57.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2fc62ed6397ec87-0062d4cf7d-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.440 DEBUG:boto:path=/mlvank-10/ 2022-07-18T03:11:57.441 DEBUG:boto:auth_path=/mlvank-10/ 2022-07-18T03:11:57.450 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.451 DEBUG:boto:Path: /mlvank-10/ 2022-07-18T03:11:57.451 DEBUG:boto:Data: 2022-07-18T03:11:57.452 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.452 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.453 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.453 DEBUG:boto:Params: {} 2022-07-18T03:11:57.472 DEBUG:boto:Token: None 2022-07-18T03:11:57.473 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/ 2022-07-18T03:11:57.474 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8= 2022-07-18T03:11:57.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8=', 'Content-Length': '0'} 2022-07-18T03:11:57.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047d4f4bb3ccbbf54-0062d4cf7d-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.505 DEBUG:boto:path=/mlvank-10/%3A 2022-07-18T03:11:57.505 DEBUG:boto:auth_path=/mlvank-10/%3A 2022-07-18T03:11:57.506 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.507 DEBUG:boto:Path: /mlvank-10/%3A 2022-07-18T03:11:57.507 DEBUG:boto:Data: 2022-07-18T03:11:57.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:57.508 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.509 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.518 DEBUG:boto:Params: {} 2022-07-18T03:11:57.519 DEBUG:boto:Token: None 2022-07-18T03:11:57.527 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/%3A 2022-07-18T03:11:57.538 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:NAuvyS23GfJzipGiYRnwfrFmDDM= 2022-07-18T03:11:57.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:NAuvyS23GfJzipGiYRnwfrFmDDM='} 2022-07-18T03:11:57.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff9b1fd64425cfd3-0062d4cf7d-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.561 DEBUG:boto:path=/mlvank-10/ 2022-07-18T03:11:57.637 DEBUG:boto:auth_path=/mlvank-10/ 2022-07-18T03:11:57.638 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.639 DEBUG:boto:Path: /mlvank-10/ 2022-07-18T03:11:57.640 DEBUG:boto:Data: 2022-07-18T03:11:57.670 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.671 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.684 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.685 DEBUG:boto:Params: {} 2022-07-18T03:11:57.705 DEBUG:boto:Token: None 2022-07-18T03:11:57.706 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/ 2022-07-18T03:11:57.719 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8= 2022-07-18T03:11:57.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:JgtkTyW2e0r48H0eqye2e1DCUe8=', 'Content-Length': '0'} 2022-07-18T03:11:57.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002b7a42e421943bb-0062d4cf7d-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.739 DEBUG:boto:path=/mlvank-10/%26 2022-07-18T03:11:57.740 DEBUG:boto:auth_path=/mlvank-10/%26 2022-07-18T03:11:57.742 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.744 DEBUG:boto:Path: /mlvank-10/%26 2022-07-18T03:11:57.745 DEBUG:boto:Data: 2022-07-18T03:11:57.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:57.747 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:11:57.748 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.749 DEBUG:boto:Params: {} 2022-07-18T03:11:57.750 DEBUG:boto:Token: None 2022-07-18T03:11:57.751 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:57 GMT /mlvank-10/%26 2022-07-18T03:11:57.757 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:g1Y8o7RSLrcWJE9mtHsfmZW7aXQ= 2022-07-18T03:11:57.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:g1Y8o7RSLrcWJE9mtHsfmZW7aXQ='} 2022-07-18T03:11:57.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e063d0d5f0cf571-0062d4cf7d-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.775 DEBUG:boto:path=/mlvank-11/ 2022-07-18T03:11:57.775 DEBUG:boto:auth_path=/mlvank-11/ 2022-07-18T03:11:57.776 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.776 DEBUG:boto:Path: /mlvank-11/ 2022-07-18T03:11:57.777 DEBUG:boto:Data: 2022-07-18T03:11:57.777 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.779 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:57.779 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.780 DEBUG:boto:Params: {} 2022-07-18T03:11:57.781 DEBUG:boto:Token: None 2022-07-18T03:11:57.782 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-11/ 2022-07-18T03:11:57.782 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:GlzW5idr6ElWg9DGXsc10G5B/J0= 2022-07-18T03:11:57.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:GlzW5idr6ElWg9DGXsc10G5B/J0=', 'Content-Length': '0'} 2022-07-18T03:11:57.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000336abe14fc20691b-0062d4cf7d-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.813 DEBUG:boto:path=/mlvank-11/myobj 2022-07-18T03:11:57.814 DEBUG:boto:auth_path=/mlvank-11/myobj 2022-07-18T03:11:57.815 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.816 DEBUG:boto:Path: /mlvank-11/myobj 2022-07-18T03:11:57.817 DEBUG:boto:Data: 2022-07-18T03:11:57.826 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:57.827 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:57.828 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.829 DEBUG:boto:Params: {} 2022-07-18T03:11:57.849 DEBUG:boto:Token: None 2022-07-18T03:11:57.850 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:57 GMT /mlvank-11/myobj 2022-07-18T03:11:57.859 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:+5btfuqMhXXoN6a6KnTQHHv7qAY= 2022-07-18T03:11:57.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:+5btfuqMhXXoN6a6KnTQHHv7qAY='} 2022-07-18T03:11:57.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008917e77e5c60fe3b-0062d4cf7d-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.888 DEBUG:boto:path=/mlvank-11/ 2022-07-18T03:11:57.889 DEBUG:boto:auth_path=/mlvank-11/ 2022-07-18T03:11:57.898 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.899 DEBUG:boto:Path: /mlvank-11/ 2022-07-18T03:11:57.900 DEBUG:boto:Data: 2022-07-18T03:11:57.901 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.904 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:57.905 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.906 DEBUG:boto:Params: {} 2022-07-18T03:11:57.907 DEBUG:boto:Token: None 2022-07-18T03:11:57.908 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-11/ 2022-07-18T03:11:57.909 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:GlzW5idr6ElWg9DGXsc10G5B/J0= 2022-07-18T03:11:57.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:GlzW5idr6ElWg9DGXsc10G5B/J0=', 'Content-Length': '0'} 2022-07-18T03:11:57.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ef0108269d840aa-0062d4cf7d-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.926 DEBUG:boto:path=/mlvank-11/_myobj 2022-07-18T03:11:57.927 DEBUG:boto:auth_path=/mlvank-11/_myobj 2022-07-18T03:11:57.928 DEBUG:boto:Method: PUT 2022-07-18T03:11:57.929 DEBUG:boto:Path: /mlvank-11/_myobj 2022-07-18T03:11:57.930 DEBUG:boto:Data: 2022-07-18T03:11:57.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:57.933 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:57.940 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.940 DEBUG:boto:Params: {} 2022-07-18T03:11:57.941 DEBUG:boto:Token: None 2022-07-18T03:11:57.942 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:57 GMT /mlvank-11/_myobj 2022-07-18T03:11:57.945 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:DgdnKDSbkBrniYWgib4Xq3EOltI= 2022-07-18T03:11:57.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:DgdnKDSbkBrniYWgib4Xq3EOltI='} 2022-07-18T03:11:57.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000823a6a91896f1d9-0062d4cf7d-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:11:57 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:57.971 DEBUG:boto:path=/mlvank-11/ 2022-07-18T03:11:57.972 DEBUG:boto:auth_path=/mlvank-11/ 2022-07-18T03:11:57.973 DEBUG:boto:Method: HEAD 2022-07-18T03:11:57.995 DEBUG:boto:Path: /mlvank-11/ 2022-07-18T03:11:57.995 DEBUG:boto:Data: 2022-07-18T03:11:57.996 DEBUG:boto:Headers: {} 2022-07-18T03:11:57.996 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:57.997 DEBUG:boto:Port: 8000 2022-07-18T03:11:57.998 DEBUG:boto:Params: {} 2022-07-18T03:11:57.999 DEBUG:boto:Token: None 2022-07-18T03:11:57.999 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:57 GMT /mlvank-11/ 2022-07-18T03:11:58.000 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:GlzW5idr6ElWg9DGXsc10G5B/J0= 2022-07-18T03:11:58.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:57 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:GlzW5idr6ElWg9DGXsc10G5B/J0=', 'Content-Length': '0'} 2022-07-18T03:11:58.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fffdbfb4d890453-0062d4cf7e-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.017 DEBUG:boto:path=/mlvank-11/%3A 2022-07-18T03:11:58.018 DEBUG:boto:auth_path=/mlvank-11/%3A 2022-07-18T03:11:58.019 DEBUG:boto:Method: PUT 2022-07-18T03:11:58.020 DEBUG:boto:Path: /mlvank-11/%3A 2022-07-18T03:11:58.021 DEBUG:boto:Data: 2022-07-18T03:11:58.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:58.023 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:58.024 DEBUG:boto:Port: 8000 2022-07-18T03:11:58.025 DEBUG:boto:Params: {} 2022-07-18T03:11:58.026 DEBUG:boto:Token: None 2022-07-18T03:11:58.028 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:58 GMT /mlvank-11/%3A 2022-07-18T03:11:58.029 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:DfCI7Emi95xXDkwg86nyr7V+jMw= 2022-07-18T03:11:58.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:DfCI7Emi95xXDkwg86nyr7V+jMw='} 2022-07-18T03:11:58.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009069ce78633fb801-0062d4cf7e-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.085 DEBUG:boto:path=/mlvank-11/ 2022-07-18T03:11:58.086 DEBUG:boto:auth_path=/mlvank-11/ 2022-07-18T03:11:58.087 DEBUG:boto:Method: HEAD 2022-07-18T03:11:58.088 DEBUG:boto:Path: /mlvank-11/ 2022-07-18T03:11:58.089 DEBUG:boto:Data: 2022-07-18T03:11:58.089 DEBUG:boto:Headers: {} 2022-07-18T03:11:58.090 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:58.092 DEBUG:boto:Port: 8000 2022-07-18T03:11:58.093 DEBUG:boto:Params: {} 2022-07-18T03:11:58.094 DEBUG:boto:Token: None 2022-07-18T03:11:58.095 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:58 GMT /mlvank-11/ 2022-07-18T03:11:58.096 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:ugC1971Iml7/3WKZ88lqBjsk4ic= 2022-07-18T03:11:58.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:ugC1971Iml7/3WKZ88lqBjsk4ic=', 'Content-Length': '0'} 2022-07-18T03:11:58.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007113c6e5ce80ab1-0062d4cf7e-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.108 DEBUG:boto:path=/mlvank-11/%26 2022-07-18T03:11:58.109 DEBUG:boto:auth_path=/mlvank-11/%26 2022-07-18T03:11:58.113 DEBUG:boto:Method: PUT 2022-07-18T03:11:58.114 DEBUG:boto:Path: /mlvank-11/%26 2022-07-18T03:11:58.115 DEBUG:boto:Data: 2022-07-18T03:11:58.164 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:58.165 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:11:58.174 DEBUG:boto:Port: 8000 2022-07-18T03:11:58.175 DEBUG:boto:Params: {} 2022-07-18T03:11:58.176 DEBUG:boto:Token: None 2022-07-18T03:11:58.177 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:58 GMT /mlvank-11/%26 2022-07-18T03:11:58.178 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:vViS+Uoe3BodhKtMt3vIGko9GCo= 2022-07-18T03:11:58.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:vViS+Uoe3BodhKtMt3vIGko9GCo='} 2022-07-18T03:11:58.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bde755bc45d135d-0062d4cf7e-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.236 DEBUG:boto:path=/mlvank-12/ 2022-07-18T03:11:58.237 DEBUG:boto:auth_path=/mlvank-12/ 2022-07-18T03:11:58.255 DEBUG:boto:Method: HEAD 2022-07-18T03:11:58.256 DEBUG:boto:Path: /mlvank-12/ 2022-07-18T03:11:58.257 DEBUG:boto:Data: 2022-07-18T03:11:58.260 DEBUG:boto:Headers: {} 2022-07-18T03:11:58.261 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.263 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.265 DEBUG:boto:Params: {} 2022-07-18T03:11:58.266 DEBUG:boto:Token: None 2022-07-18T03:11:58.267 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/ 2022-07-18T03:11:58.268 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc= 2022-07-18T03:11:58.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc=', 'Content-Length': '0'} 2022-07-18T03:11:58.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a7fba2a0530b190-0062d4cf7e-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.303 DEBUG:boto:path=/mlvank-12/myobj 2022-07-18T03:11:58.304 DEBUG:boto:auth_path=/mlvank-12/myobj 2022-07-18T03:11:58.304 DEBUG:boto:Method: PUT 2022-07-18T03:11:58.305 DEBUG:boto:Path: /mlvank-12/myobj 2022-07-18T03:11:58.314 DEBUG:boto:Data: 2022-07-18T03:11:58.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:58.315 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.316 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.317 DEBUG:boto:Params: {} 2022-07-18T03:11:58.325 DEBUG:boto:Token: None 2022-07-18T03:11:58.326 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/myobj 2022-07-18T03:11:58.327 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:zJrZ+zp7uxrTssR3s1LXMZEyyVk= 2022-07-18T03:11:58.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:zJrZ+zp7uxrTssR3s1LXMZEyyVk='} 2022-07-18T03:11:58.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d437c891e3ccc53-0062d4cf7e-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.341 DEBUG:boto:path=/mlvank-12/ 2022-07-18T03:11:58.342 DEBUG:boto:auth_path=/mlvank-12/ 2022-07-18T03:11:58.343 DEBUG:boto:Method: HEAD 2022-07-18T03:11:58.345 DEBUG:boto:Path: /mlvank-12/ 2022-07-18T03:11:58.346 DEBUG:boto:Data: 2022-07-18T03:11:58.347 DEBUG:boto:Headers: {} 2022-07-18T03:11:58.349 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.351 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.352 DEBUG:boto:Params: {} 2022-07-18T03:11:58.352 DEBUG:boto:Token: None 2022-07-18T03:11:58.353 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/ 2022-07-18T03:11:58.354 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc= 2022-07-18T03:11:58.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc=', 'Content-Length': '0'} 2022-07-18T03:11:58.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b1416ffdb9465e5-0062d4cf7e-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.384 DEBUG:boto:path=/mlvank-12/_myobj 2022-07-18T03:11:58.385 DEBUG:boto:auth_path=/mlvank-12/_myobj 2022-07-18T03:11:58.385 DEBUG:boto:Method: PUT 2022-07-18T03:11:58.386 DEBUG:boto:Path: /mlvank-12/_myobj 2022-07-18T03:11:58.387 DEBUG:boto:Data: 2022-07-18T03:11:58.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:58.389 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.411 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.411 DEBUG:boto:Params: {} 2022-07-18T03:11:58.412 DEBUG:boto:Token: None 2022-07-18T03:11:58.412 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/_myobj 2022-07-18T03:11:58.422 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2iTR6/n6q3C/AaCElMRQpBMxQ+s= 2022-07-18T03:11:58.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2iTR6/n6q3C/AaCElMRQpBMxQ+s='} 2022-07-18T03:11:58.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cde072eaddb2dc98-0062d4cf7e-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.465 DEBUG:boto:path=/mlvank-12/ 2022-07-18T03:11:58.466 DEBUG:boto:auth_path=/mlvank-12/ 2022-07-18T03:11:58.483 DEBUG:boto:Method: HEAD 2022-07-18T03:11:58.484 DEBUG:boto:Path: /mlvank-12/ 2022-07-18T03:11:58.499 DEBUG:boto:Data: 2022-07-18T03:11:58.502 DEBUG:boto:Headers: {} 2022-07-18T03:11:58.504 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.505 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.505 DEBUG:boto:Params: {} 2022-07-18T03:11:58.506 DEBUG:boto:Token: None 2022-07-18T03:11:58.507 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/ 2022-07-18T03:11:58.507 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc= 2022-07-18T03:11:58.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc=', 'Content-Length': '0'} 2022-07-18T03:11:58.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c6e7efa62ec9b60-0062d4cf7e-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.526 DEBUG:boto:path=/mlvank-12/%3A 2022-07-18T03:11:58.526 DEBUG:boto:auth_path=/mlvank-12/%3A 2022-07-18T03:11:58.527 DEBUG:boto:Method: PUT 2022-07-18T03:11:58.528 DEBUG:boto:Path: /mlvank-12/%3A 2022-07-18T03:11:58.528 DEBUG:boto:Data: 2022-07-18T03:11:58.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:58.553 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.554 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.555 DEBUG:boto:Params: {} 2022-07-18T03:11:58.555 DEBUG:boto:Token: None 2022-07-18T03:11:58.556 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/%3A 2022-07-18T03:11:58.557 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:j2/5koOh5Dhhhy/zDnZKsBjDS5I= 2022-07-18T03:11:58.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:j2/5koOh5Dhhhy/zDnZKsBjDS5I='} 2022-07-18T03:11:58.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cc032f9c9a519d1-0062d4cf7e-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.573 DEBUG:boto:path=/mlvank-12/ 2022-07-18T03:11:58.577 DEBUG:boto:auth_path=/mlvank-12/ 2022-07-18T03:11:58.578 DEBUG:boto:Method: HEAD 2022-07-18T03:11:58.579 DEBUG:boto:Path: /mlvank-12/ 2022-07-18T03:11:58.579 DEBUG:boto:Data: 2022-07-18T03:11:58.580 DEBUG:boto:Headers: {} 2022-07-18T03:11:58.586 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.587 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.587 DEBUG:boto:Params: {} 2022-07-18T03:11:58.588 DEBUG:boto:Token: None 2022-07-18T03:11:58.589 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/ 2022-07-18T03:11:58.590 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc= 2022-07-18T03:11:58.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2OVugKKUCHPX2NkHNQQFtfgVNZc=', 'Content-Length': '0'} 2022-07-18T03:11:58.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f3d3e6a1ea6550b-0062d4cf7e-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.603 DEBUG:boto:path=/mlvank-12/%26 2022-07-18T03:11:58.604 DEBUG:boto:auth_path=/mlvank-12/%26 2022-07-18T03:11:58.605 DEBUG:boto:Method: PUT 2022-07-18T03:11:58.620 DEBUG:boto:Path: /mlvank-12/%26 2022-07-18T03:11:58.621 DEBUG:boto:Data: 2022-07-18T03:11:58.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:11:58.623 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:11:58.624 DEBUG:boto:Port: 8001 2022-07-18T03:11:58.625 DEBUG:boto:Params: {} 2022-07-18T03:11:58.636 DEBUG:boto:Token: None 2022-07-18T03:11:58.637 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:11:58 GMT /mlvank-12/%26 2022-07-18T03:11:58.638 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:I+FmH47aHmAo428rFBHZ+8UDbd0= 2022-07-18T03:11:58.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:11:58 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:I+FmH47aHmAo428rFBHZ+8UDbd0='} 2022-07-18T03:11:58.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000846a6687393b0b63-0062d4cf7e-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:11:58 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:11:58.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:58.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:58.652 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:58.709 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:58.710 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:58.754 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:11:58.772 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:11:58.773 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:11:58.773 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:58.774 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:11:58.775 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:11:58.775 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:11:58.776 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:11:58.776 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:11:58.789 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:11:58.790 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:11:58.794 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:11:58.794 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:11:57.162796Z" 2022-07-18T03:11:58.795 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:11:58.796 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:11:58.797 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:11:58.797 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:11:58.798 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:11:58.799 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:11:58.799 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:58.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:11:58.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:58.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:11:58.816 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:11:58.880 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:11:58.882 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:11:58.991 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:11:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:11:58.994 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:11:58.994 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:11:58.995 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:11:58.995 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:11:58.996 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:58.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:59.005 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:11:59.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:59.006 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:11:59.007 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:59.009 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:59.010 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:11:59.020 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:59.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:11:59.021 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:59.043 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:11:59.047 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:59.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:59.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:59.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:59.058 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:11:59.058 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:59.059 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:59.059 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:11:59.060 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:59.060 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:59.061 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:59.061 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:11:59.062 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:59.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:59.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:59.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:59.064 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:11:59.064 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:59.065 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:59.100 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.078377_81.1", 2022-07-18T03:11:59.101 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:59.101 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:59.102 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:59.103 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.078377Z", 2022-07-18T03:11:59.103 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:59.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:59.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:59.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:11:59.105 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:11:59.106 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:11:59.106 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:11:59.107 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:11:59.107 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:11:59.108 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:11:59.108 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:11:59.109 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:11:59.109 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:11:59.109 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:59.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:11:59.129 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:11:59.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:59.130 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:11:59.130 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:11:59.131 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:11:59.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:11:59.132 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:11:58Z" 2022-07-18T03:11:59.150 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:11:59.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.078377_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.078377Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:11:58Z" } 2022-07-18T03:11:59.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.078377_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.078377Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:11:59.160 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:11:59.175 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.078377_81.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:11:59.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658113917.162796_88.1 target=1_1658113917.078377_81.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:04.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:04.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:04.204 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:04.256 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:04.257 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:04.333 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:04.334 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:04.334 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:04.335 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:04.335 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:04.337 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:04.338 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:04.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:04.339 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:04.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:04.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:04.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:04.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:04.343 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:04.343 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:04.350 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:04.351 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:04.352 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:04.352 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:04.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:04.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:04.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:04.355 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:04.355 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:04.356 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:04.356 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:04.357 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:04.357 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:04.358 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:04.358 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:04.359 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:04.359 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:04.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:04.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:04.362 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:04.362 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:04.363 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:04.363 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:04.364 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:04.364 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:04.365 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:04.365 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:04.365 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:04.366 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:04.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:04.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:04.367 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:04.368 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:04.368 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:04.368 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:04.369 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:04.369 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:04.370 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:04.370 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:04.370 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:04.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:04.371 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:04.371 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:04.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:04.371 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:04.372 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:04.372 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:04.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:04.374 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:04Z" 2022-07-18T03:12:04.375 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:04.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:04Z" } 2022-07-18T03:12:04.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:04.376 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:04.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:04.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:09.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:09.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:09.383 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:09.442 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:09.443 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:09.528 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:09.529 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:09.529 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:09.530 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:09.531 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:09.531 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:09.532 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:09.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:09.532 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:09.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:09.533 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:09.534 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:09.534 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:09.535 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:09.535 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:09.536 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:09.537 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:09.538 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:09.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:09.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:09.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:09.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:09.540 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:09.541 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:09.542 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:09.542 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:09.543 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:09.544 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:09.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:09.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:09.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:09.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:09.547 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:09.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:09.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:09.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:09.550 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:09.550 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:09.551 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:09.552 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:09.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:09.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:09.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:09.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:09.556 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:09.556 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:09.557 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:09.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:09.558 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:09.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:09.559 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:09.559 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:09.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:09.561 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:09.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:09.562 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:09.562 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:09.563 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:09.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:09.564 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:09Z" 2022-07-18T03:12:09.564 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:09.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:09Z" } 2022-07-18T03:12:09.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:09.567 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:09.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:09.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:14.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:14.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:14.574 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:14.629 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:14.630 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:14.679 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:14.680 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:14.680 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:14.681 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:14.681 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:14.682 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:14.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:14.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:14.682 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:14.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:14.683 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:14.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:14.683 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:14.684 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:14.684 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:14.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:14.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:14.685 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:14.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:14.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:14.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:14.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:14.686 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:14.687 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:14.687 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:14.688 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:14.688 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:14.688 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:14.689 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:14.689 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:14.689 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:14.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:14.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:14.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:14.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:14.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:14.703 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:14.703 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:14.704 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:14.705 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:14.706 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:14.707 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:14.708 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:14.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:14.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:14.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:14.710 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:14.711 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:14.714 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:14.715 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:14.715 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:14.716 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:14.716 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:14.717 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:14.718 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:14.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:14.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:14.719 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:14.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:14.720 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:14.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:14.722 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:14.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:14.725 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:14Z" 2022-07-18T03:12:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:14.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:14Z" } 2022-07-18T03:12:14.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:14.738 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:14.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:14.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:19.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:19.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:19.765 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:19.814 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:19.814 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:19.880 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:19.881 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:19.882 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:19.882 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:19.883 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:19.883 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:19.884 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:19.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:19.885 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:19.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:19.886 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:19.886 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:19.886 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:19.891 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:19.891 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:19.892 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:19.892 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:19.892 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:19.892 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:19.893 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:19.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:19.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:19.893 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:19.894 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:19.894 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:19.895 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:19.895 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:19.895 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:19.896 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:19.896 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:19.897 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:19.897 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:19.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:19.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:19.898 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:19.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:19.898 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:19.899 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:19.899 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:19.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:19.900 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:19.900 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:19.900 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:19.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:19.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:19.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:19.905 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:19.906 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:19.906 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:19.907 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:19.908 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:19.908 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:19.909 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:19.910 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:19.910 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:19.911 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:19.911 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:19.911 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:19.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:19.912 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:19.913 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:19.914 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:19.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:19.914 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:19Z" 2022-07-18T03:12:19.915 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:19.919 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:19Z" } 2022-07-18T03:12:19.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:19.921 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:19.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:19.923 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:24.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:24.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:24.927 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:24.994 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:24.995 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:25.164 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:25.164 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:25.165 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:25.166 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:25.166 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:25.166 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:25.166 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:25.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:25.167 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:25.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:25.167 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:25.168 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:25.168 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:25.169 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:25.169 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:25.169 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:25.170 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:25.170 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:25.170 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:25.170 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:25.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:25.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:25.171 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:25.171 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:25.172 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:25.173 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:25.173 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:25.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:25.174 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:25.174 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:25.174 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:25.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:25.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:25.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:25.175 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:25.179 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:25.179 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:25.180 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:25.180 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:25.180 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:25.180 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:25.181 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:25.182 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:25.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:25.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:25.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:25.183 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:25.184 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:25.185 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:25.185 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:25.185 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:25.185 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:25.186 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:25.186 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:25.186 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:25.186 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:25.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:25.187 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:25.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:25.187 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:25.188 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:25.188 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:25.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:25.188 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:25Z" 2022-07-18T03:12:25.189 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:25.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:25Z" } 2022-07-18T03:12:25.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:25.202 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:25.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:25.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:30.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:30.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:30.210 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:30.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:30.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:30.323 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:30.325 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:30.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:30.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:30.325 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:30.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:30.326 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:30.331 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:30.344 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:30.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:30.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:30.376 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:30.376 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:30.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:30.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:30.377 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:30.377 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:30.377 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:30.382 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:30.383 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:30.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:30.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:30.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:30.386 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:30.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:30.388 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:30.388 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:30.389 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:30.389 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:30.389 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:30.390 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:30.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:30.415 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:30.415 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:30Z" 2022-07-18T03:12:30.443 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:30.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:30Z" } 2022-07-18T03:12:30.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:30.449 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:30.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:30.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:35.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:35.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:35.456 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:35.510 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:35.511 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:35.582 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:35.583 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:35.583 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:35.584 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:35.584 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:35.585 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:35.585 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:35.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:35.586 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:35.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:35.587 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:35.587 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:35.588 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:35.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:35.589 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:35.589 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:35.590 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:35.590 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:35.591 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:35.591 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:35.593 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:35.593 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:35.594 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:35.594 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:35.595 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:35.595 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:35.597 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:35.597 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:35.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:35.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:35.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:35.600 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:35.600 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:35.601 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:35.601 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:35.602 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:35.602 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:35.603 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:35.603 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:35.604 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:35.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:35.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:35.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:35.606 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:35.607 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:35.607 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:35.617 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:35.618 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:35.618 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:35.619 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:35.619 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:35.620 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:35.620 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:35.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:35.621 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:35.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:35.622 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:35.623 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:35.623 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:35.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:35.624 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:35Z" 2022-07-18T03:12:35.625 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:35.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:35Z" } 2022-07-18T03:12:35.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:35.649 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:35.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:35.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:40.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:40.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:40.662 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:40.712 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:40.712 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:40.782 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:40.782 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:40.782 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:40.783 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:40.783 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:40.783 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:40.783 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:40.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:40.784 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:40.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:40.784 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:40.785 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:40.785 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:40.785 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:40.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:40.786 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:40.786 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:40.786 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:40.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:40.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:40.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:40.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:40.787 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:40.788 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:40.788 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:40.788 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:40.788 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:40.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:40.789 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:40.789 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:40.794 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:40.794 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:40.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:40.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:40.795 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:40.795 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:40.795 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:40.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:40.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:40.799 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:40.799 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:40.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:40.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:40.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:40.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:40.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:40.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:40.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:40.804 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:40.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:40Z" 2022-07-18T03:12:40.805 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:40.806 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:40Z" } 2022-07-18T03:12:40.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:40.807 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:40.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:40.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:45.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:45.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:45.812 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:45.886 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:45.899 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:45.970 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:45.971 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:45.971 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:45.972 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:45.972 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:45.973 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:45.973 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:45.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:45.975 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:45.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:45.976 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:45.976 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:45.977 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:45.981 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:45.982 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:45.983 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:45.983 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:45.984 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:45.985 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:45.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:45.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:45.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:45.987 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:45.988 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:45.988 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:45.989 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:45.989 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:45.990 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:45.990 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:45.991 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:45.992 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:45.992 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:45.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:45.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:45.994 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:45.994 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:45.995 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:45.996 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:45.996 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:45.997 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:45.997 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:45.997 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:45.998 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:45.999 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:45.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:45.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:46.000 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:46.001 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:46.001 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:46.002 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:46.002 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:46.002 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:46.003 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:46.003 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:46.004 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:46.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:46.005 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:46.005 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:46.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:46.007 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:46.008 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:46.008 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:46.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:46.011 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:45Z" 2022-07-18T03:12:46.011 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:46.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:45Z" } 2022-07-18T03:12:46.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:46.013 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:46.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:46.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:51.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:51.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:51.017 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:51.075 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:51.075 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:51.150 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:51.151 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:51.151 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:51.151 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:51.151 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:51.152 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:51.152 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:51.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:51.152 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:51.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:51.155 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:51.155 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:51.155 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:51.156 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:51.156 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:51.156 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:51.157 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:51.157 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:51.157 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:51.157 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:51.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:51.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:51.158 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:51.158 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:51.159 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:51.159 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:51.159 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:51.159 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:51.160 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:51.160 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:51.160 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:51.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:51.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:51.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:51.161 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:51.162 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:51.162 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:51.162 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:51.162 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:51.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:51.163 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:51.163 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:51.163 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:51.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:51.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:51.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:51.167 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:51.168 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:51.168 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:51.169 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:51.169 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:51.169 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:51.170 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:51.170 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:51.170 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:51.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:51.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:51.171 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:51.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:51.172 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:51.172 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:51.172 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:51.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:51.173 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:51Z" 2022-07-18T03:12:51.173 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:51.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:51Z" } 2022-07-18T03:12:51.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:51.181 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:51.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:51.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:12:56.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:56.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:12:56.186 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:12:56.243 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:12:56.244 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:12:56.308 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:12:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:12:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:12:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:12:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:12:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:12:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:12:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:12:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:12:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:12:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:12:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:12:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:12:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:12:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:12:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:12:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:12:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:12:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:12:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:12:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:12:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:12:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:12:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:12:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:12:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:12:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:12:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:12:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:12:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:12:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:12:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:12:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:12:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:12:56Z" 2022-07-18T03:12:56.350 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:12:56.353 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:12:56Z" } 2022-07-18T03:12:56.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:12:56.356 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:12:56.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:12:56.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:01.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:01.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:01.366 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:01.420 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:01.421 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:01.485 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:01.485 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:01.486 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:01.487 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:01.488 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:01.488 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:01.489 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:01.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:01.490 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:01.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:01.491 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:01.491 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:01.492 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:01.492 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:01.493 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:01.494 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:01.494 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:01.494 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:01.495 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:01.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:01.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:01.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:01.498 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:01.498 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:01.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:01.499 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:01.499 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:01.500 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:01.500 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:01.501 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:01.501 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:01.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:01.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:01.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:01.504 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:01.504 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:01.505 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:01.505 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:01.506 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:01.506 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:01.507 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:01.507 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:01.509 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:01.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:01.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:01.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:01.511 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:01.511 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:01.512 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:01.512 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:01.513 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:01.514 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:01.515 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:01.516 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:01.516 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:01.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:01.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:01.517 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:01.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:01.518 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:01.518 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:01.518 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:01.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:01.519 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:01Z" 2022-07-18T03:13:01.519 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:01.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:01Z" } 2022-07-18T03:13:01.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:01.523 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:01.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:01.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:06.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:06.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:06.529 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:06.583 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:06.584 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:06.643 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:06.644 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:06.644 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:06.645 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:06.645 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:06.646 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:06.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:06.647 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:06.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:06.648 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:06.648 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:06.649 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:06.649 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:06.650 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:06.652 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:06.652 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:06.653 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:06.653 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:06.654 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:06.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:06.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:06.655 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:06.656 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:06.658 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:06.658 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:06.659 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:06.659 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:06.660 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:06.660 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:06.661 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:06.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:06.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:06.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:06.663 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:06.663 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:06.664 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:06.665 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:06.666 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:06.667 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:06.668 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:06.669 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:06.669 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:06.670 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:06.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:06.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:06.671 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:06.672 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:06.672 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:06.673 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:06.674 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:06.674 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:06.675 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:06.675 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:06.675 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:06.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:06.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:06.676 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:06.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:06.677 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:06.677 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:06.677 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:06.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:06.678 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:06Z" 2022-07-18T03:13:06.678 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:06.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:06Z" } 2022-07-18T03:13:06.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:06.681 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:06.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:06.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:11.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:11.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:11.687 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:11.738 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:11.739 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:11.799 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:11.800 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:11.800 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:11.800 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:11.801 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:11.801 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:11.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:11.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:11.802 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:11.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:11.809 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:11.809 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:11.809 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:11.810 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:11.810 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:11.810 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:11.810 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:11.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:11.812 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:11.812 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:11.812 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:11.813 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:11.813 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:11.813 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:11.814 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:11.814 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:11.814 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:11.814 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:11.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:11.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:11.815 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:11.815 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:11.817 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:11.817 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:11.817 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:11.818 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:11.818 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:11.818 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:11.818 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:11.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:11.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:11.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:11.819 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:11.820 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:11.820 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:11.820 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:11.820 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:11.821 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:11.821 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:11.821 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:11.821 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:11.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:11.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:11.823 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:11.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:11.823 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:11.823 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:11.824 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:11.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:11.824 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:11Z" 2022-07-18T03:13:11.824 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:11.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:11Z" } 2022-07-18T03:13:11.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:11.833 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:11.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:11.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:16.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:16.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:16.838 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:16.890 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:16.891 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:16.957 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:16.958 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:16.958 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:16.958 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:16.961 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:16.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:16.962 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:16.962 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:16.962 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:16.964 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:16.964 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:16.967 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:16.968 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:16.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:16.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:16.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:16.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:16.972 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:16.973 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:16.973 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:16.974 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:16.974 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:16.975 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:16.975 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:16.976 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:16.977 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:16.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:16.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:16.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:16.979 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:16.980 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:16.981 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:16.981 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:16.983 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:16.983 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:16.984 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:16.984 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:16.985 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:16.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:16.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:16.986 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:16.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:16.987 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:16.988 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:16.989 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:16.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:16.990 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:16Z" 2022-07-18T03:13:16.990 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:16.992 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:16Z" } 2022-07-18T03:13:16.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:16.995 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:16.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:16.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:21.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:22.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:22.001 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:22.055 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:22.057 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:22.128 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:22.129 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:22.131 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:22.131 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:22.132 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:22.132 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:22.133 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:22.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:22.134 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:22.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:22.135 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:22.135 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:22.136 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:22.137 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:22.137 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:22.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:22.138 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:22.139 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:22.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:22.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:22.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:22.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:22.142 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:22.142 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:22.143 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:22.143 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:22.144 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:22.145 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:22.145 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:22.146 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:22.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:22.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:22.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:22.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:22.150 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:22.150 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:22.151 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:22.153 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:22.153 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:22.154 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:22.154 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:22.155 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:22.155 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:22.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:22.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:22.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:22.157 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:22.157 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:22Z" 2022-07-18T03:13:22.169 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:22.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:22Z" } 2022-07-18T03:13:22.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:22.174 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:22.175 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:22.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:27.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:27.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:27.180 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:27.242 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:27.243 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:27.299 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:27.300 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:27.300 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:27.301 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:27.301 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:27.302 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:27.302 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:27.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:27.306 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:27.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:27.307 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:27.307 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:27.308 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:27.309 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:27.310 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:27.311 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:27.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:27.312 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:27.312 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:27.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:27.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:27.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:27.314 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:27.315 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:27.315 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:27.316 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:27.316 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:27.316 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:27.317 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:27.317 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:27.319 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:27.320 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:27.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:27.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:27.321 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:27.321 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:27.322 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:27.322 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:27Z" 2022-07-18T03:13:27.341 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:27.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:27Z" } 2022-07-18T03:13:27.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:27.344 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:27.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:27.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:32.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:32.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:32.350 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:32.406 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:32.406 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:32.488 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:32.488 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:32.489 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:32.489 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:32.489 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:32.490 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:32.490 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:32.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:32.491 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:32.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:32.491 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:32.492 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:32.492 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:32.492 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:32.503 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:32.503 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:32.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:32.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:32.505 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:32.506 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:32.507 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:32.507 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:32.508 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:32.508 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:32.508 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:32.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:32.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:32.514 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:32.520 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:32.521 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:32.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:32.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:32.522 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:32.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:32.522 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:32.523 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:32Z" 2022-07-18T03:13:32.524 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:32.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:32Z" } 2022-07-18T03:13:32.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:32.526 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:32.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:32.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:37.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:37.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:37.531 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:37.594 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:37.595 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:37.665 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:37.669 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:37.669 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:37.671 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:37.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:37.672 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:37.673 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:37.674 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:37.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:37.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:37.677 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:37.677 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:37.679 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:37.679 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:37.680 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:37.680 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:37.681 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:37.681 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:37.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:37.682 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:37.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:37.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:37.684 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:37.684 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:37.684 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:37.685 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:37.685 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:37.686 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:37.686 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:37.686 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:37.687 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:37.687 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:37.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:37.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:37.688 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:37.688 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:37.689 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:37.689 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:37.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:37.689 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:37.690 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:37.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:37.690 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:37.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:37.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:37.691 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:37.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:37.692 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:37.692 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:37.692 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:37.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:37.692 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:37Z" 2022-07-18T03:13:37.693 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:37.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:37Z" } 2022-07-18T03:13:37.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:37.695 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:37.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:37.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:42.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:42.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:42.699 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:42.752 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:42.753 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:42.840 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:42.840 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:42.841 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:42.841 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:42.842 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:42.843 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:42.844 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:42.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:42.845 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:42.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:42.846 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:42.847 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:42.847 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:42.848 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:42.848 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:42.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:42.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:42.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:42.851 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:42.852 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:42.852 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:42.853 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:42.853 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:42.853 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:42.854 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:42.854 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:42.854 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:42.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:42.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:42.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:42.855 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:42.856 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:42.856 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:42.856 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:42.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:42.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:42.857 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:42.857 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:42.857 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:42.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:42.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:42.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:42.859 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:42.859 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:42.860 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:42.860 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:42.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:42.861 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:42.861 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:42.861 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:42.861 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:42.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:42.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:42.863 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:42.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:42.863 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:42.864 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:42.864 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:42.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:42.865 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:42Z" 2022-07-18T03:13:42.866 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:42.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:42Z" } 2022-07-18T03:13:42.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:42.870 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:42.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:42.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:47.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:47.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:47.877 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:47.928 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:47.929 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:48.001 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:48.003 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:48.003 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:48.004 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:48.007 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:48.008 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:48.008 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:48.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:48.009 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:48.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:48.010 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:48.011 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:48.011 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:48.012 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:48.012 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:48.013 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:48.013 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:48.015 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:48.015 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:48.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:48.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:48.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:48.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:48.018 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:48.018 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:48.019 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:48.019 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:48.020 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:48.020 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:48.021 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:48.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:48.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:48.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:48.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:48.024 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:48.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:48.025 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:48.026 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:48.026 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:48.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:48.027 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:48.028 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:48.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:48.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:48.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:48.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:48.030 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:48.031 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:48.032 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:48.032 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:48.033 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:48.033 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:48.034 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:48.034 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:48.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:48.035 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:48.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:48.036 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:48.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:48.037 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:48.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:48.038 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:48.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:48.044 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:47Z" 2022-07-18T03:13:48.045 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:48.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:47Z" } 2022-07-18T03:13:48.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:48.050 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:48.051 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:48.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:53.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:53.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:53.056 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:53.106 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:53.107 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:53.187 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:53.187 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:53.188 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:53.188 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:53.188 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:53.188 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:53.189 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:53.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:53.189 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:53.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:53.190 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:53.190 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:53.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:53.191 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:53.191 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:53.191 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:53.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:53.192 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:53.192 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:53.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:53.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:53.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:53.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:53.196 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:53.196 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:53.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:53.197 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:53.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:53.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:53.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:53.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:53.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:53.202 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:53.202 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:53.203 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:53.203 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:53.204 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:53.204 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:53.204 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:53.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:53.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:53.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:53.206 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:53.207 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:53.207 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:53.208 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:53.208 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:53.208 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:53.209 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:53.209 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:53.210 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:53.210 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:53.210 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:53.211 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:53.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:53.211 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:53.211 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:53.212 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:53.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:53.212 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:53Z" 2022-07-18T03:13:53.212 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:53.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:53Z" } 2022-07-18T03:13:53.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:53.216 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:53.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:53.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:13:58.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:58.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:13:58.222 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:13:58.275 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:13:58.276 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:13:58.334 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:13:58.334 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:13:58.335 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:13:58.335 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:13:58.335 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:13:58.335 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:13:58.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:58.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:58.336 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:13:58.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:58.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:13:58.338 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:58.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:58.339 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:13:58.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:58.339 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:13:58.340 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:58.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:13:58.341 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:58.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:58.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:58.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:58.348 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:13:58.349 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:58.349 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:58.350 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:13:58.352 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:58.353 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:58.353 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:58.354 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:13:58.354 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:58.356 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:58.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:58.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:58.357 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:13:58.359 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:58.363 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:58.363 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:13:58.364 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:58.364 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:58.364 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:58.364 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:13:58.365 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:58.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:58.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:58.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:13:58.366 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:13:58.366 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:13:58.366 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:13:58.367 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:13:58.367 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:13:58.367 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:13:58.367 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:13:58.368 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:13:58.368 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:13:58.368 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:58.368 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:13:58.369 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:13:58.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:58.369 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:13:58.369 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:13:58.370 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:13:58.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:13:58.370 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:13:58Z" 2022-07-18T03:13:58.370 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:13:58.372 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:13:58Z" } 2022-07-18T03:13:58.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:13:58.373 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:13:58.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:13:58.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:03.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:03.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:03.381 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:03.443 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:03.443 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:03.516 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:03.517 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:03.517 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:03.518 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:03.518 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:03.519 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:03.519 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:03.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:03.520 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:03.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:03.521 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:03.522 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:03.522 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:03.523 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:03.523 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:03.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:03.524 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:03.525 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:03.526 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:03.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:03.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:03.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:03.529 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:03.529 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:03.530 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:03.531 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:03.531 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:03.531 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:03.532 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:03.532 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:03.533 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:03.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:03.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:03.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:03.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:03.536 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:03.536 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:03.537 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:03.537 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:03.538 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:03.539 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:03.539 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:03.539 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:03.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:03.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:03.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:03.541 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:03.542 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:03.542 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:03.543 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:03.543 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:03.544 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:03.544 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:03.545 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:03.545 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:03.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:03.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:03.547 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:03.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:03.548 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:03.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:03.549 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:03.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:03.550 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:03Z" 2022-07-18T03:14:03.550 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:03.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:03Z" } 2022-07-18T03:14:03.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:03.554 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:03.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:03.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:08.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:08.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:08.559 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:08.625 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:08.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:08.767 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:08.768 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:08.768 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:08.769 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:08.769 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:08.770 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:08.770 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:08.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:08.771 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:08.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:08.773 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:08.773 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:08.774 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:08.774 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:08.775 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:08.775 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:08.775 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:08.776 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:08.776 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:08.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:08.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:08.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:08.778 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:08.779 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:08.779 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:08.780 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:08.780 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:08.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:08.782 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:08.783 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:08.783 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:08.784 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:08.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:08.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:08.785 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:08.785 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:08.786 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:08.786 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:08.787 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:08.787 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:08.788 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:08.788 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:08.789 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:08.789 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:08.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:08.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:08.791 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:08.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:08.792 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:08.792 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:08.792 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:08.793 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:08.793 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:08.794 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:08.794 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:08.794 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:08.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:08.795 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:08.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:08.796 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:08.796 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:08.797 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:08.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:08.798 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:08Z" 2022-07-18T03:14:08.798 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:08.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:08Z" } 2022-07-18T03:14:08.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:08.801 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:08.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:08.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:13.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:13.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:13.808 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:13.870 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:13.871 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:13.942 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:13.942 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:13.943 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:13.943 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:13.943 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:13.947 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:13.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:13.947 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:13.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:13.959 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:13.960 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:13.960 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:13.961 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:13.961 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:13.962 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:13.962 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:13.963 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:13.964 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:13.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:13.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:13.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:13.967 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:13.967 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:13.968 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:13.968 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:13.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:13.969 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:13.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:13.973 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:13.974 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:13.975 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:13.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:13.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:13.979 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:13.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:13.980 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:13.980 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:13.981 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:13.981 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:13.982 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:13.982 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:13.983 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:13.989 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:13.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:13.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:13.990 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:13.990 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:13.991 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:13.991 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:13.991 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:13.992 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:13.992 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:13.992 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:13.992 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:13.997 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:13.998 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:13.998 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:13.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:13.999 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:14.000 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:14.000 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:14.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:14.001 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:13Z" 2022-07-18T03:14:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:14.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:13Z" } 2022-07-18T03:14:14.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:14.006 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:14.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:14.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:19.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:19.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:19.014 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:19.078 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:19.078 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:19.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:19.145 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:19.145 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:19.146 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:19.147 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:19.148 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:19.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:19.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:19.149 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:19.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:19.150 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:19.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:19.151 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:19.152 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:19.152 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:19.153 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:19.153 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:19.153 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:19.153 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:19.154 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:19.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:19.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:19.155 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:19.155 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:19.156 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:19.156 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:19.157 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:19.158 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:19.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:19.159 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:19.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:19.160 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:19.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:19.162 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:19.162 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:19.162 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:19.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:19.164 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:19.165 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:19.165 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:19.166 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:19.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:19.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:19.168 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:19.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:19.173 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:19Z" 2022-07-18T03:14:19.173 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:19.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:19Z" } 2022-07-18T03:14:19.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:19.175 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:19.175 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:19.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:24.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:24.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:24.180 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:24.239 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:24.240 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:24.303 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:24.304 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:24.304 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:24.305 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:24.305 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:24.306 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:24.306 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:24.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:24.307 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:24.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:24.308 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:24.308 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:24.309 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:24.309 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:24.309 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:24.310 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:24.310 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:24.310 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:24.311 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:24.311 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:24.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:24.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:24.312 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:24.312 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:24.314 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:24.314 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:24.315 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:24.315 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:24.318 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:24.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:24.319 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:24.319 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:24.320 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:24.320 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:24.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:24.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:24.321 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:24.323 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:24.323 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:24.323 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:24.324 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:24.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:24.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:24.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:24.327 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:24.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:24.330 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:24Z" 2022-07-18T03:14:24.330 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:24.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:24Z" } 2022-07-18T03:14:24.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:24.334 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:24.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:24.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:29.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:29.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:29.339 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:29.394 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:29.395 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:29.496 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:29.497 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:29.497 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:29.497 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:29.498 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:29.498 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:29.498 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:29.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:29.499 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:29.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:29.499 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:29.500 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:29.500 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:29.500 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:29.500 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:29.501 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:29.501 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:29.501 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:29.502 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:29.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:29.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:29.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:29.503 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:29.503 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:29.503 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:29.503 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:29.504 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:29.504 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:29.504 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:29.505 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:29.505 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:29.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:29.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:29.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:29.507 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:29.507 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:29.507 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:29.508 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:29.508 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:29.508 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:29.508 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:29.509 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:29.509 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:29.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:29.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:29.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:29.510 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:29.510 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:29.510 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:29.511 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:29.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:29.511 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:29.511 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:29.512 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:29.512 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:29.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:29.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:29.513 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:29.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:29.513 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:29.514 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:29.514 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:29.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:29.515 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:29Z" 2022-07-18T03:14:29.515 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:29.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:29Z" } 2022-07-18T03:14:29.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:29.517 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:29.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:29.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:34.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:34.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:34.524 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:34.582 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:34.583 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:34.658 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:34.659 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:34.659 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:34.659 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:34.659 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:34.660 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:34.660 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:34.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:34.661 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:34.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:34.663 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:34.664 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:34.664 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:34.664 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:34.665 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:34.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:34.665 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:34.665 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:34.666 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:34.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:34.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:34.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:34.667 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:34.667 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:34.667 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:34.667 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:34.668 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:34.668 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:34.668 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:34.668 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:34.669 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:34.669 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:34.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:34.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:34.672 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:34.672 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:34.673 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:34.673 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:34.674 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:34.674 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:34.675 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:34.675 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:34.676 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:34.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:34.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:34.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:34.677 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:34.678 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:34.678 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:34.679 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:34.679 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:34.679 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:34.680 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:34.680 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:34.681 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:34.682 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:34.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:34.683 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:34.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:34.686 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:34.686 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:34.687 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:34.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:34.692 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:34Z" 2022-07-18T03:14:34.692 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:34.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:34Z" } 2022-07-18T03:14:34.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:34.708 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:34.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:34.709 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:39.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:39.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:39.712 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:39.774 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:39.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:39.923 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:39.924 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:39.925 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:39.925 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:39.926 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:39.926 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:39.926 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:39.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:39.927 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:39.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:39.927 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:39.929 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:39.929 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:39.929 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:39.930 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:39.930 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:39.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:39.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:39Z" 2022-07-18T03:14:39.941 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:39.942 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:39Z" } 2022-07-18T03:14:39.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:39.943 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:39.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:39.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:44.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:44.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:44.948 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:45.006 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:45.007 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:45.084 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:45.085 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:45.085 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:45.088 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:45.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:45.089 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:45.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:45.090 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:45.091 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:45.091 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:45.091 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:45.093 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:45.093 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:45.093 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:45.094 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:45.094 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:45.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:45.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:45.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:45.096 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:45.096 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:45.097 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:45.097 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:45.098 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:45.098 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:45.099 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:45.099 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:45.099 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:45.100 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:45.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:45.102 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:45.103 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:45.105 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:45.105 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:45.106 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:45.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:45.108 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:45.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:45.111 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:45.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:45Z" 2022-07-18T03:14:45.113 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:45.116 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:45Z" } 2022-07-18T03:14:45.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:45.118 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:45.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:45.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:50.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:50.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:50.126 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:50.182 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:50.183 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:50.252 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:50.257 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:50.258 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:50.258 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:50.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:50.266 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:50.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:50.267 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:50.268 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:50.268 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:50.269 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:50.269 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:50.270 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:50.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:50.280 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:50.282 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:50.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:50.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:50.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:50.288 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:50.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:50.289 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:50.289 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:50.291 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:50.292 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:50.294 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:50.295 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:50.296 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:50.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:50.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:50.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:50.298 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:50.298 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:50.299 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:50.299 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:50.300 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:50.300 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:50.301 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:50.301 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:50.306 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:50.307 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:50.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:50.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:50.309 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:50.309 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:50.310 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:50.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:50.310 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:50.310 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:50.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:50.311 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:50.311 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:50.311 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:50.312 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:50.312 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:50.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:50.312 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:50.313 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:50.313 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:50.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:50.315 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:50Z" 2022-07-18T03:14:50.316 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:50.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:50Z" } 2022-07-18T03:14:50.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:50.320 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:50.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:50.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:14:55.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:55.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:14:55.325 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:14:55.384 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:14:55.385 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:14:55.457 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:14:55.458 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:14:55.463 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:14:55.463 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:14:55.463 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:14:55.463 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:14:55.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:55.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:55.465 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:14:55.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:55.466 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:14:55.466 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:55.466 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:55.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:14:55.467 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:55.468 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:14:55.468 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:55.468 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:14:55.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:55.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:55.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:55.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:55.471 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:14:55.471 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:55.471 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:55.472 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:14:55.472 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:55.472 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:55.473 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:55.473 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:14:55.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:55.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:55.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:55.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:55.474 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:14:55.474 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:55.475 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:55.475 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:14:55.476 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:55.476 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:55.477 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:55.477 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:14:55.477 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:55.477 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:55.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:55.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:14:55.478 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:14:55.479 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:14:55.479 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:14:55.479 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:14:55.479 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:14:55.481 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:14:55.482 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:14:55.482 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:14:55.483 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:14:55.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:55.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:14:55.483 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:14:55.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:55.484 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:14:55.484 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:14:55.484 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:14:55.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:14:55.485 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:14:55Z" 2022-07-18T03:14:55.486 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:14:55.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:14:55Z" } 2022-07-18T03:14:55.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:14:55.487 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:14:55.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:14:55.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:00.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:00.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:00.492 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:00.549 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:00.549 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:00.612 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:00.613 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:00.613 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:00.614 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:00.615 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:00.615 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:00.615 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:00.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:00.616 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:00.616 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:00.616 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:00.617 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:00.617 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:00.618 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:00.618 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:00.618 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:00.618 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:00.619 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:00.619 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:00.620 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:00.620 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:00.620 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:00.621 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:00.621 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:00.622 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:00.622 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:00.623 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:00.623 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:00.624 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:00.624 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:00.624 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:00.625 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:00.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:00.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:00.626 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:00.626 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:00.626 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:00.626 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:00.627 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:00.627 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:00.627 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:00.627 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:00.628 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:00.628 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:00.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:00.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:00.629 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:00.629 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:00.629 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:00.630 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:00.630 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:00.630 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:00.630 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:00.631 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:00.631 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:00.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:00.632 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:00.632 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:00.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:00.632 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:00.633 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:00.633 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:00.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:00.634 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:00Z" 2022-07-18T03:15:00.634 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:00.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:00Z" } 2022-07-18T03:15:00.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:00.637 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:00.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:00.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:05.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:05.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:05.661 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:05.711 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:05.713 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:05.783 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:05.783 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:05.784 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:05.784 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:05.785 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:05.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:05.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:05.787 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:05.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:05.788 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:05.789 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:05.790 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:05.790 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:05.791 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:05.791 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:05.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:05.792 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:05.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:05.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:05.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:05.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:05.795 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:05.795 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:05.796 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:05.796 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:05.796 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:05.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:05.797 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:05.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:05.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:05.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:05.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:05.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:05.800 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:05.801 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:05.801 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:05.802 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:05.802 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:05.803 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:05.803 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:05.823 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:05.823 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:05.824 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:05.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:05.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:05.833 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:05.833 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:05.837 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:05.838 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:05.839 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:05.839 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:05.868 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:05.870 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:05.870 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:05.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:05.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:05.872 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:05.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:05.873 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:05.873 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:05.874 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:05.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:05.875 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:05Z" 2022-07-18T03:15:05.875 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:05.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:05Z" } 2022-07-18T03:15:05.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:05.889 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:05.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:05.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:10.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:10.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:10.905 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:10.972 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:10.973 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:11.060 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:11.060 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:11.061 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:11.061 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:11.065 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:11.072 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:11.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:11.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:11.074 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:11.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:11.074 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:11.075 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:11.075 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:11.075 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:11.075 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:11.076 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:11.076 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:11.076 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:11.077 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:11.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:11.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:11.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:11.078 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:11.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:11.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:11.078 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:11.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:11.079 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:11.079 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:11.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:11.081 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:11.081 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:11.082 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:11.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:11.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:11.083 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:11.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:11.083 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:11.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:11.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:11.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:11.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:11.085 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:11.085 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:11.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:11.086 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:11.086 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:11.086 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:11.087 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:11.087 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:11.087 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:11.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:11.088 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:11.088 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:11.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:11.088 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:11.089 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:11.089 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:11.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:11.089 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:11Z" 2022-07-18T03:15:11.090 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:11.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:11Z" } 2022-07-18T03:15:11.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:11.092 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:11.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:11.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:16.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:16.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:16.099 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:16.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:16.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:16.213 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:16.214 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:16.214 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:16.215 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:16.215 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:16.216 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:16.216 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:16.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:16.217 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:16.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:16.218 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:16.219 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:16.219 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:16.220 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:16.220 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:16.221 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:16.221 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:16.222 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:16.222 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:16.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:16.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:16.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:16.224 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:16.225 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:16.225 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:16.225 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:16.226 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:16.226 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:16.226 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:16.227 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:16.227 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:16.227 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:16.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:16.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:16.229 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:16.229 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:16.229 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:16.230 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:16.231 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:16.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:16.231 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:16.232 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:16.232 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:16.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:16.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:16.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:16.234 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:16.234 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:16.235 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:16.235 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:16.235 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:16.236 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:16.236 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:16.236 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:16.237 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:16.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:16.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:16.237 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:16.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:16.238 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:16.238 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:16.238 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:16.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:16.239 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:16Z" 2022-07-18T03:15:16.239 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:16.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:16Z" } 2022-07-18T03:15:16.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:16.243 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:16.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:16.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:21.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:21.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:21.258 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:21.309 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:21.309 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:21.376 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:21.376 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:21.377 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:21.377 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:21.377 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:21.377 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:21.378 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:21.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:21.378 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:21.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:21.379 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:21.379 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:21.380 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:21.380 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:21.380 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:21.381 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:21.381 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:21.382 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:21.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:21.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:21.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:21.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:21.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:21.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:21.384 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:21.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:21.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:21.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:21.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:21.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:21.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:21.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:21.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:21.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:21.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:21.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:21.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:21.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:21.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:21.388 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:21.389 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:21.389 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:21.389 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:21.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:21.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:21.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:21.390 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:21.390 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:21.391 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:21.392 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:21.392 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:21.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:21.393 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:21.393 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:21.393 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:21.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:21.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:21.394 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:21.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:21.395 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:21.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:21.395 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:21.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:21.396 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:21Z" 2022-07-18T03:15:21.396 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:21.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:21Z" } 2022-07-18T03:15:21.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:21.397 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:21.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:21.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:26.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:26.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:26.404 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:26.460 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:26.461 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:26.544 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:26.544 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:26.545 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:26.545 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:26.545 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:26.545 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:26.546 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:26.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:26.546 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:26.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:26.547 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:26.547 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:26.547 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:26.548 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:26.548 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:26.548 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:26.549 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:26.549 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:26.549 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:26.549 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:26.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:26.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:26.550 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:26.550 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:26.551 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:26.551 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:26.551 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:26.553 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:26.553 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:26.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:26.554 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:26.554 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:26.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:26.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:26.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:26.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:26.555 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:26.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:26.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:26.556 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:26.557 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:26.557 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:26.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:26.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:26.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:26.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:26.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:26.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:26.559 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:26.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:26.560 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:26.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:26.560 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:26.560 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:26.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:26.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:26.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:26.562 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:26.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:26.563 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:26.563 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:26.564 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:26.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:26.564 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:26Z" 2022-07-18T03:15:26.565 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:26.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:26Z" } 2022-07-18T03:15:26.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:26.567 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:26.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:26.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:31.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:31.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:31.573 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:31.622 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:31.623 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:31.688 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:31.689 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:31.689 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:31.690 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:31.690 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:31.691 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:31.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:31.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:31.692 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:31.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:31.693 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:31.693 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:31.694 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:31.694 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:31.695 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:31.695 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:31.696 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:31.696 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:31.696 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:31.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:31.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:31.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:31.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:31.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:31.700 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:31.701 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:31.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:31.702 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:31.706 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:31.706 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:31.706 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:31.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:31.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:31.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:31.707 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:31.708 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:31.708 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:31.708 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:31.708 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:31.709 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:31.709 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:31.709 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:31.711 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:31.711 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:31.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:31.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:31.714 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:31.716 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:31.716 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:31.717 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:31.717 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:31.718 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:31.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:31.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:31.719 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:31.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:31.720 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:31.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:31.721 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:31.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:31.722 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:31Z" 2022-07-18T03:15:31.723 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:31.725 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:31Z" } 2022-07-18T03:15:31.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:31.727 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:31.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:31.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:36.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:36.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:36.734 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:36.787 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:36.788 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:36.872 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:36.873 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:36.873 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:36.882 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:36.883 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:36.883 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:36.884 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:36.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:36.885 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:36.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:36.886 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:36.886 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:36.887 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:36.887 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:36.888 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:36.888 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:36.889 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:36.889 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:36.889 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:36.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:36.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:36.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:36.903 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:36.904 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:36.904 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:36.905 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:36.905 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:36.906 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:36.907 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:36.907 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:36.907 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:36.908 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:36.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:36.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:36.909 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:36.910 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:36.910 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:36.910 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:36.911 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:36.912 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:36.912 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:36.912 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:36.913 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:36.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:36.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:36.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:36.916 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:36.916 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:36.916 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:36.916 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:36.917 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:36.917 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:36.917 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:36.918 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:36.918 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:36.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:36.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:36.919 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:36.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:36.920 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:36.920 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:36.920 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:36.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:36.921 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:36Z" 2022-07-18T03:15:36.922 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:36.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:36Z" } 2022-07-18T03:15:36.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:36.925 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:36.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:36.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:41.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:41.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:41.931 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:41.986 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:41.989 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:42.053 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:42.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:42.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:42.078 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:42.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:42.079 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:42.079 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:42.080 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:42.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:42.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:42.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:42.081 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:42.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:42.081 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:42.082 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:42.082 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:42.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:42.083 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:42Z" 2022-07-18T03:15:42.083 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:42.084 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:42Z" } 2022-07-18T03:15:42.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:42.085 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:42.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:42.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:47.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:47.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:47.091 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:47.166 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:47.167 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:47.325 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:47.325 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:47.325 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:47.327 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:47.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:47.327 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:47.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:47.328 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:47.328 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:47.329 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:47.329 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:47.329 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:47.331 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:47.332 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:47.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:47.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:47.333 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:47.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:47.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:47.334 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:47.334 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:47.335 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:47.335 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:47.336 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:47.336 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:47.337 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:47.337 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:47.338 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:47.338 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:47.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:47.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:47.339 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:47.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:47.340 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:47.340 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:47.340 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:47.341 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:47.341 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:47.341 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:47.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:47.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:47.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:47.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:47.345 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:47.346 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:47.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:47.347 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:47.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:47.348 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:47.349 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:47.349 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:47.350 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:47.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:47.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:47.351 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:47.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:47.352 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:47.354 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:47.355 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:47.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:47.356 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:47Z" 2022-07-18T03:15:47.356 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:47.357 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:47Z" } 2022-07-18T03:15:47.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:47.359 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:47.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:47.361 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:52.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:52.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:52.367 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:52.419 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:52.419 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:52.482 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:52.483 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:52.483 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:52.484 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:52.484 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:52.485 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:52.485 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:52.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:52.495 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:52.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:52.496 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:52.496 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:52.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:52.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:52.499 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:52.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:52.500 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:52.500 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:52.500 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:52.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:52.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:52.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:52.502 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:52.503 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:52.503 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:52.504 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:52.504 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:52.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:52.505 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:52.519 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:52.520 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:52.520 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:52.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:52.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:52.522 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:52.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:52.523 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:52.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:52.527 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:52.528 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:52.528 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:52.529 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:52.529 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:52.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:52.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:52.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:52.533 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:52.533 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:52.534 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:52.534 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:52.535 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:52.535 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:52.536 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:52.536 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:52.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:52.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:52.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:52.538 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:52.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:52.539 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:52.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:52.540 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:52.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:52.546 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:52Z" 2022-07-18T03:15:52.547 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:52.547 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:52Z" } 2022-07-18T03:15:52.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:52.550 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:52.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:52.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:15:57.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:57.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:15:57.573 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:15:57.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:15:57.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:15:57.678 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:15:57.679 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:15:57.679 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:15:57.679 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:15:57.680 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:15:57.680 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:15:57.680 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:57.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:57.681 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:15:57.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:57.681 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:15:57.681 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:57.682 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:57.682 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:15:57.682 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:57.682 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:15:57.688 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:57.688 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:15:57.688 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:57.688 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:57.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:57.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:57.689 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:15:57.689 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:57.690 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:57.690 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:15:57.690 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:57.690 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:57.691 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:57.691 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:15:57.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:57.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:57.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:57.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:57.692 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:15:57.692 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:57.693 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:57.693 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:15:57.696 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:57.696 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:57.698 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:57.699 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:15:57.699 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:57.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:57.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:57.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:15:57.701 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:15:57.702 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:15:57.702 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:15:57.702 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:15:57.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:15:57.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:15:57.704 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:15:57.704 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:15:57.704 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:15:57.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:57.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:15:57.707 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:15:57.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:57.708 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:15:57.709 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:15:57.709 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:15:57.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:15:57.710 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:15:57Z" 2022-07-18T03:15:57.711 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:15:57.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:15:57Z" } 2022-07-18T03:15:57.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:15:57.715 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:15:57.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:15:57.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:02.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:02.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:02.718 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:02.774 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:02.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:02.849 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:02.849 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:02.849 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:02.849 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:02.850 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:02.850 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:02.851 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:02.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:02.851 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:02.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:02.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:02.853 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:02.853 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:02.854 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:02.854 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:02.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:02.854 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:02.855 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:02.855 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:02.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:02.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:02.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:02.856 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:02.857 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:02.857 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:02.857 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:02.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:02.858 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:02.858 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:02.858 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:02.859 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:02.859 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:02.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:02.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:02.859 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:02.860 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:02.860 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:02.860 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:02.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:02.861 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:02.861 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:02.862 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:02.862 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:02.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:02.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:02.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:02.864 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:02.865 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:02.865 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:02.866 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:02.866 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:02.867 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:02.868 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:02.868 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:02.868 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:02.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:02.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:02.870 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:02.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:02.873 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:02.873 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:02.874 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:02.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:02.875 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:02Z" 2022-07-18T03:16:02.876 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:02.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:02Z" } 2022-07-18T03:16:02.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:02.880 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:02.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:02.881 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:07.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:07.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:07.886 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:07.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:07.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:08.071 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:08.071 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:08.072 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:08.073 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:08.073 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:08.074 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:08.074 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:08.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:08.076 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:08.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:08.077 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:08.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:08.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:08.079 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:08.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:08.079 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:08.081 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:08.081 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:08.082 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:08.082 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:08.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:08.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:08.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:08.084 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:08.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:08.087 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:08.088 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:08.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:08.089 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:08.089 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:08.090 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:08.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:08.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:08.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:08.091 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:08.092 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:08.092 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:08.093 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:08.093 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:08.094 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:08.094 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:08.096 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:08.098 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:08.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:08.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:08.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:08.100 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:08.100 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:08.101 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:08.101 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:08.102 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:08.102 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:08.103 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:08.103 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:08.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:08.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:08.105 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:08.105 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:08.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:08.107 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:08.107 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:08.110 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:08.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:08.112 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:08Z" 2022-07-18T03:16:08.112 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:08.115 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:08Z" } 2022-07-18T03:16:08.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:08.118 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:08.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:08.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:13.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:13.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:13.124 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:13.181 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:13.181 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:13.256 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:13.257 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:13.258 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:13.258 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:13.259 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:13.259 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:13.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:13.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:13.260 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:13.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:13.261 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:13.261 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:13.262 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:13.262 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:13.263 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:13.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:13.264 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:13.264 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:13.265 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:13.265 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:13.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:13.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:13.267 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:13.267 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:13.268 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:13.268 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:13.268 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:13.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:13.269 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:13.270 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:13.270 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:13.270 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:13.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:13.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:13.271 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:13.271 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:13.272 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:13.272 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:13.273 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:13.274 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:13.274 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:13.274 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:13.274 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:13.275 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:13.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:13.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:13.275 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:13.276 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:13.276 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:13.276 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:13.276 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:13.277 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:13.277 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:13.278 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:13.278 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:13.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:13.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:13.279 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:13.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:13.279 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:13.279 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:13.280 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:13.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:13.280 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:13Z" 2022-07-18T03:16:13.280 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:13.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:13Z" } 2022-07-18T03:16:13.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:13.284 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:13.284 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:13.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:18.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:18.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:18.287 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:18.337 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:18.337 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:18.388 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:18.389 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:18.389 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:18.390 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:18.390 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:18.390 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:18.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:18.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:18.391 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:18.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:18.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:18.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:18.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:18.393 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:18.393 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:18.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:18.394 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:18.395 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:18.395 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:18.396 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:18.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:18.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:18.397 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:18.397 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:18.397 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:18.398 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:18.398 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:18.398 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:18.398 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:18.399 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:18.399 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:18.399 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:18.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:18.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:18.400 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:18.400 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:18.401 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:18.401 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:18.401 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:18.402 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:18.402 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:18.402 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:18.402 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:18.403 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:18.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:18.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:18.403 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:18.404 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:18.404 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:18.404 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:18.404 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:18.405 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:18.405 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:18.405 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:18.405 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:18.406 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:18.406 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:18.406 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:18.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:18.407 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:18.407 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:18.407 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:18.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:18.408 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:18Z" 2022-07-18T03:16:18.408 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:18.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:18Z" } 2022-07-18T03:16:18.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:18.411 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:18.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:18.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:23.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:23.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:23.415 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:23.465 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:23.465 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:23.530 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:23.530 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:23.530 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:23.531 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:23.531 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:23.531 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:23.532 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:23.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:23.532 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:23.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:23.533 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:23.533 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:23.533 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:23.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:23.534 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:23.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:23.535 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:23.535 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:23.536 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:23.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:23.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:23.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:23.537 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:23.538 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:23.538 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:23.538 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:23.539 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:23.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:23.539 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:23.540 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:23.540 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:23.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:23.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:23.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:23.541 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:23.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:23.542 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:23.543 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:23.543 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:23.544 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:23.544 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:23.545 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:23.545 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:23.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:23.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:23.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:23.547 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:23.547 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:23.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:23.548 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:23.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:23.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:23.550 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:23.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:23.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:23.551 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:23.551 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:23.551 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:23.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:23.552 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:23.552 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:23.552 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:23.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:23.553 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:23Z" 2022-07-18T03:16:23.554 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:23.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:23Z" } 2022-07-18T03:16:23.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:23.557 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:23.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:23.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:28.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:28.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:28.564 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:28.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:28.627 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:28.708 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:28Z" 2022-07-18T03:16:28.741 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:28.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:28Z" } 2022-07-18T03:16:28.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:28.744 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:28.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:28.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:33.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:33.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:33.751 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:33.806 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:33.806 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:33.881 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:33.882 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:33.882 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:33.882 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:33.883 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:33.883 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:33.883 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:33.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:33.884 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:33.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:33.884 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:33.885 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:33.885 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:33.885 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:33.886 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:33.886 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:33.886 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:33.887 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:33.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:33.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:33.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:33.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:33.888 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:33.888 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:33.889 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:33.889 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:33.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:33.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:33.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:33.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:33.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:33.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:33.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:33.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:33.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:33.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:33.892 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:33.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:33.893 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:33.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:33.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:33.894 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:33.894 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:33.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:33.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:33.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:33.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:33.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:33.896 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:33.896 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:33.897 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:33.897 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:33.897 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:33.897 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:33.898 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:33.898 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:33.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:33.899 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:33.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:33.900 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:33.901 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:33.901 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:33.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:33.902 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:33Z" 2022-07-18T03:16:33.902 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:33.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:33Z" } 2022-07-18T03:16:33.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:33.904 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:33.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:33.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:38.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:38.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:38.913 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:38.971 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:38.971 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:39.031 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:39.032 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:39.033 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:39.034 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:39.034 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:39.034 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:39.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:39.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:39.035 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:39.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:39.036 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:39.036 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:39.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:39.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:39.038 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:39.039 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:39.039 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:39.040 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:39.040 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:39.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:39.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:39.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:39.042 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:39.043 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:39.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:39.044 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:39.045 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:39.045 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:39.046 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:39.046 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:39.047 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:39.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:39.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:39.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:39.049 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:39.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:39.050 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:39.051 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:39.052 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:39.052 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:39.053 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:39.053 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:39.054 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:39.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:39.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:39.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:39.057 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:39.057 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:39.058 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:39.059 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:39.059 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:39.060 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:39.060 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:39.061 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:39.062 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:39.062 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:39.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:39.063 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:39.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:39.064 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:39.065 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:39.065 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:39.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:39.066 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:39Z" 2022-07-18T03:16:39.066 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:39.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:39Z" } 2022-07-18T03:16:39.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:39.068 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:39.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:39.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:44.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:44.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:44.075 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:44.135 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:44.136 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:44.214 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:44.215 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:44.215 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:44.216 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:44.216 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:44.216 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:44.217 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:44.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:44.219 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:44.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:44.220 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:44.220 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:44.221 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:44.221 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:44.221 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:44.222 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:44.222 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:44.222 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:44.223 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:44.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:44.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:44.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:44.226 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:44.227 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:44.227 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:44.227 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:44.227 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:44.228 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:44.228 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:44.228 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:44.228 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:44.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:44.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:44.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:44.229 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:44.230 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:44.230 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:44.230 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:44.231 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:44.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:44.231 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:44.231 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:44.231 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:44.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:44.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:44.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:44.232 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:44.233 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:44.233 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:44.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:44.233 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:44.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:44.234 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:44.234 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:44.234 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:44.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:44.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:44.235 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:44.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:44.236 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:44.237 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:44.237 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:44.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:44.237 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:44Z" 2022-07-18T03:16:44.238 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:44.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:44Z" } 2022-07-18T03:16:44.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:44.239 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:44.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:44.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:49.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:49.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:49.243 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:49.295 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:49.296 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:49.370 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:49.372 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:49.372 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:49.372 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:49.373 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:49.373 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:49.373 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:49.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:49.378 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:49.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:49.379 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:49.379 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:49.380 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:49.380 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:49.381 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:49.381 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:49.382 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:49.382 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:49.383 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:49.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:49.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:49.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:49.387 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:49.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:49.388 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:49.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:49.389 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:49.389 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:49.390 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:49.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:49.390 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:49.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:49.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:49.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:49.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:49.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:49.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:49.394 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:49.394 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:49.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:49.395 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:49.395 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:49.396 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:49.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:49.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:49.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:49.399 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:49.400 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:49.400 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:49.401 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:49.401 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:49.402 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:49.402 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:49.403 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:49.403 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:49.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:49.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:49.404 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:49.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:49.405 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:49.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:49.406 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:49.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:49.408 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:49Z" 2022-07-18T03:16:49.408 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:49.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:49Z" } 2022-07-18T03:16:49.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:49.413 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:49.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:49.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:54.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:54.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:54.417 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:54.472 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:54.473 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:54.546 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:54.546 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:54.547 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:54.547 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:54.547 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:54.548 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:54.548 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:54.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:54.548 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:54.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:54.549 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:54.549 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:54.549 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:54.550 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:54.550 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:54.550 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:54.551 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:54.551 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:54.551 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:54.551 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:54.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:54.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:54.552 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:54.553 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:54.553 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:54.554 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:54.554 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:54.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:54.555 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:54.557 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:54.558 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:54.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:54.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:54.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:54.560 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:54.560 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:54.560 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:54.561 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:54.561 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:54.562 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:54.563 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:54.564 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:54.564 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:54.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:54.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:54.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:54.566 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:54.566 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:54.568 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:54.568 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:54.569 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:54.569 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:54.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:54.570 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:54.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:54.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:54.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:54.572 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:54.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:54.574 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:54.574 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:54.575 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:54.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:54.575 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:54Z" 2022-07-18T03:16:54.575 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:54.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:54Z" } 2022-07-18T03:16:54.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:54.577 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:54.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:54.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:16:59.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:59.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:16:59.585 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:16:59.642 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:16:59.643 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:16:59.707 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:16:59.707 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:16:59.708 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:16:59.708 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:16:59.708 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:16:59.709 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:16:59.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:59.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:59.710 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:16:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:16:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:16:59.714 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:59.714 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:16:59.715 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:59.716 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:16:59.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:59.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:59.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:59.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:59.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:16:59.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:59.718 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:59.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:16:59.718 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:59.719 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:59.719 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:59.719 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:16:59.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:59.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:59.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:59.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:59.721 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:16:59.722 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:59.722 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:59.723 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:16:59.723 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:59.723 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:59.723 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:59.724 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:16:59.724 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:59.724 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:59.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:59.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:16:59.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:16:59.725 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:16:59.725 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:16:59.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:16:59.726 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:16:59.726 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:16:59.726 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:16:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:16:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:16:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:16:59.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:16:59.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:59.728 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:16:59.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:16:59.729 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:16:59.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:16:59.730 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:16:59Z" 2022-07-18T03:16:59.731 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:16:59.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:16:59Z" } 2022-07-18T03:16:59.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:16:59.733 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:16:59.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:16:59.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:04.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:04.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:04.737 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:04.793 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:04.793 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:04.857 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:04.858 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:04.858 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:04.859 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:04.859 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:04.860 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:04.861 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:04.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:04.862 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:04.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:04.863 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:04.864 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:04.864 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:04.864 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:04.864 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:04.865 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:04.865 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:04.865 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:04.866 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:04.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:04.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:04.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:04.867 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:04.867 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:04.867 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:04.867 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:04.868 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:04.868 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:04.868 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:04.869 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:04.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:04.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:04.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:04.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:04.870 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:04.870 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:04.871 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:04.871 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113917.162796_88.1", 2022-07-18T03:17:04.871 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:04.871 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:04.872 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:04.872 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:11:57.162796Z", 2022-07-18T03:17:04.872 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:04.872 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:04.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:04.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:04.873 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:04.874 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:04.874 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:04.874 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:04.874 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:04.875 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:04.875 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:04.875 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:04.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:04.877 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:04.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:04.877 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:04.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:04.878 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:04Z" 2022-07-18T03:17:04.878 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:04.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658113917.162796_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:11:57.162796Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:04Z" } 2022-07-18T03:17:04.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658113917.162796_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:11:57.162796Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:04.881 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:04.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658113917.162796_88.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:04.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:09.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... FAIL 2022-07-18T03:17:09.893 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-13 2022-07-18T03:17:09.894 DEBUG:boto:path=/mlvank-13/ 2022-07-18T03:17:09.894 DEBUG:boto:auth_path=/mlvank-13/ 2022-07-18T03:17:09.895 DEBUG:boto:Method: PUT 2022-07-18T03:17:09.896 DEBUG:boto:Path: /mlvank-13/ 2022-07-18T03:17:09.896 DEBUG:boto:Data: 2022-07-18T03:17:09.897 DEBUG:boto:Headers: {} 2022-07-18T03:17:09.898 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:17:09.898 DEBUG:boto:Port: 8000 2022-07-18T03:17:09.899 DEBUG:boto:Params: {} 2022-07-18T03:17:09.900 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:17:09.901 DEBUG:boto:Token: None 2022-07-18T03:17:09.902 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:17:09 GMT /mlvank-13/ 2022-07-18T03:17:09.903 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:czFnCPQnMoYD9IdEjjQ8fq+0kA4= 2022-07-18T03:17:09.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:17:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:czFnCPQnMoYD9IdEjjQ8fq+0kA4=', 'Content-Length': '0'} 2022-07-18T03:17:09.924 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b625aaaced3ca508-0062d4d0b5-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:17:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:09.924 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-14 2022-07-18T03:17:09.925 DEBUG:boto:path=/mlvank-14/ 2022-07-18T03:17:09.925 DEBUG:boto:auth_path=/mlvank-14/ 2022-07-18T03:17:09.926 DEBUG:boto:Method: PUT 2022-07-18T03:17:09.926 DEBUG:boto:Path: /mlvank-14/ 2022-07-18T03:17:09.927 DEBUG:boto:Data: 2022-07-18T03:17:09.927 DEBUG:boto:Headers: {} 2022-07-18T03:17:09.928 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:17:09.929 DEBUG:boto:Port: 8000 2022-07-18T03:17:09.929 DEBUG:boto:Params: {} 2022-07-18T03:17:09.930 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:17:09.930 DEBUG:boto:Token: None 2022-07-18T03:17:09.931 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:17:09 GMT /mlvank-14/ 2022-07-18T03:17:09.932 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:stanAIGkrHFfWUbVLLRzuHHdHYA= 2022-07-18T03:17:09.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:17:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:stanAIGkrHFfWUbVLLRzuHHdHYA=', 'Content-Length': '0'} 2022-07-18T03:17:09.967 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f0de3e535a2c2c1-0062d4d0b5-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:17:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:09.968 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-15 2022-07-18T03:17:09.968 DEBUG:boto:path=/mlvank-15/ 2022-07-18T03:17:09.969 DEBUG:boto:auth_path=/mlvank-15/ 2022-07-18T03:17:09.969 DEBUG:boto:Method: PUT 2022-07-18T03:17:09.970 DEBUG:boto:Path: /mlvank-15/ 2022-07-18T03:17:09.971 DEBUG:boto:Data: 2022-07-18T03:17:09.971 DEBUG:boto:Headers: {} 2022-07-18T03:17:09.972 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:17:09.972 DEBUG:boto:Port: 8001 2022-07-18T03:17:09.973 DEBUG:boto:Params: {} 2022-07-18T03:17:09.974 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:17:09.975 DEBUG:boto:Token: None 2022-07-18T03:17:09.975 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:17:09 GMT /mlvank-15/ 2022-07-18T03:17:09.976 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:U4SMaxXVd3cXpnCzKwcquVZOlRg= 2022-07-18T03:17:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:17:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:U4SMaxXVd3cXpnCzKwcquVZOlRg=', 'Content-Length': '0'} 2022-07-18T03:17:10.007 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5a5d9c060613aa8-0062d4d0b5-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.008 DEBUG:boto:path=/mlvank-13/ 2022-07-18T03:17:10.008 DEBUG:boto:auth_path=/mlvank-13/ 2022-07-18T03:17:10.009 DEBUG:boto:Method: HEAD 2022-07-18T03:17:10.009 DEBUG:boto:Path: /mlvank-13/ 2022-07-18T03:17:10.010 DEBUG:boto:Data: 2022-07-18T03:17:10.010 DEBUG:boto:Headers: {} 2022-07-18T03:17:10.011 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:17:10.011 DEBUG:boto:Port: 8000 2022-07-18T03:17:10.012 DEBUG:boto:Params: {} 2022-07-18T03:17:10.012 DEBUG:boto:Token: None 2022-07-18T03:17:10.013 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:17:10 GMT /mlvank-13/ 2022-07-18T03:17:10.013 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:fHAQTHpOoql9URRs4+JE5WbD3+c= 2022-07-18T03:17:10.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:17:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:fHAQTHpOoql9URRs4+JE5WbD3+c=', 'Content-Length': '0'} 2022-07-18T03:17:10.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ad5f46f3dee13e8-0062d4d0b6-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.020 DEBUG:boto:path=/mlvank-13/myobj 2022-07-18T03:17:10.020 DEBUG:boto:auth_path=/mlvank-13/myobj 2022-07-18T03:17:10.021 DEBUG:boto:Method: PUT 2022-07-18T03:17:10.022 DEBUG:boto:Path: /mlvank-13/myobj 2022-07-18T03:17:10.022 DEBUG:boto:Data: 2022-07-18T03:17:10.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:17:10.023 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:17:10.024 DEBUG:boto:Port: 8000 2022-07-18T03:17:10.024 DEBUG:boto:Params: {} 2022-07-18T03:17:10.025 DEBUG:boto:Token: None 2022-07-18T03:17:10.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:17:10 GMT /mlvank-13/myobj 2022-07-18T03:17:10.026 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:RTQpKnEg0M7iQ7j0/BDSzR9Vvgc= 2022-07-18T03:17:10.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:17:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:RTQpKnEg0M7iQ7j0/BDSzR9Vvgc='} 2022-07-18T03:17:10.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f69bc39c7e236fd4-0062d4d0b6-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.046 DEBUG:boto:path=/mlvank-14/ 2022-07-18T03:17:10.047 DEBUG:boto:auth_path=/mlvank-14/ 2022-07-18T03:17:10.048 DEBUG:boto:Method: HEAD 2022-07-18T03:17:10.049 DEBUG:boto:Path: /mlvank-14/ 2022-07-18T03:17:10.050 DEBUG:boto:Data: 2022-07-18T03:17:10.051 DEBUG:boto:Headers: {} 2022-07-18T03:17:10.052 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:17:10.052 DEBUG:boto:Port: 8000 2022-07-18T03:17:10.053 DEBUG:boto:Params: {} 2022-07-18T03:17:10.054 DEBUG:boto:Token: None 2022-07-18T03:17:10.055 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:17:10 GMT /mlvank-14/ 2022-07-18T03:17:10.056 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Q4vKdFcS+VI4v8xbg6jssb08QEY= 2022-07-18T03:17:10.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:17:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Q4vKdFcS+VI4v8xbg6jssb08QEY=', 'Content-Length': '0'} 2022-07-18T03:17:10.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083c21fceec662fe1-0062d4d0b6-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.065 DEBUG:boto:path=/mlvank-14/myobj 2022-07-18T03:17:10.065 DEBUG:boto:auth_path=/mlvank-14/myobj 2022-07-18T03:17:10.066 DEBUG:boto:Method: PUT 2022-07-18T03:17:10.067 DEBUG:boto:Path: /mlvank-14/myobj 2022-07-18T03:17:10.068 DEBUG:boto:Data: 2022-07-18T03:17:10.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:17:10.069 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:17:10.071 DEBUG:boto:Port: 8000 2022-07-18T03:17:10.072 DEBUG:boto:Params: {} 2022-07-18T03:17:10.073 DEBUG:boto:Token: None 2022-07-18T03:17:10.074 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:17:10 GMT /mlvank-14/myobj 2022-07-18T03:17:10.075 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:BPyb5BsnaQ06FV75XS06eNKGTFg= 2022-07-18T03:17:10.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:17:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:BPyb5BsnaQ06FV75XS06eNKGTFg='} 2022-07-18T03:17:10.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003920789221168fbe-0062d4d0b6-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.097 DEBUG:boto:path=/mlvank-15/ 2022-07-18T03:17:10.097 DEBUG:boto:auth_path=/mlvank-15/ 2022-07-18T03:17:10.098 DEBUG:boto:Method: HEAD 2022-07-18T03:17:10.100 DEBUG:boto:Path: /mlvank-15/ 2022-07-18T03:17:10.100 DEBUG:boto:Data: 2022-07-18T03:17:10.101 DEBUG:boto:Headers: {} 2022-07-18T03:17:10.101 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:17:10.102 DEBUG:boto:Port: 8001 2022-07-18T03:17:10.103 DEBUG:boto:Params: {} 2022-07-18T03:17:10.103 DEBUG:boto:Token: None 2022-07-18T03:17:10.104 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:17:10 GMT /mlvank-15/ 2022-07-18T03:17:10.105 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:BM7i9G8kbgGomUDovav+WfAfqbA= 2022-07-18T03:17:10.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:17:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:BM7i9G8kbgGomUDovav+WfAfqbA=', 'Content-Length': '0'} 2022-07-18T03:17:10.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f85cd034c9631f36-0062d4d0b6-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.123 DEBUG:boto:path=/mlvank-15/myobj 2022-07-18T03:17:10.123 DEBUG:boto:auth_path=/mlvank-15/myobj 2022-07-18T03:17:10.124 DEBUG:boto:Method: PUT 2022-07-18T03:17:10.125 DEBUG:boto:Path: /mlvank-15/myobj 2022-07-18T03:17:10.125 DEBUG:boto:Data: 2022-07-18T03:17:10.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-07-18T03:17:10.126 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:17:10.127 DEBUG:boto:Port: 8001 2022-07-18T03:17:10.127 DEBUG:boto:Params: {} 2022-07-18T03:17:10.128 DEBUG:boto:Token: None 2022-07-18T03:17:10.129 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 18 Jul 2022 03:17:10 GMT /mlvank-15/myobj 2022-07-18T03:17:10.129 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:oV3OkOT9rBzjGkxoyLohHP80TYQ= 2022-07-18T03:17:10.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:17:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:oV3OkOT9rBzjGkxoyLohHP80TYQ='} 2022-07-18T03:17:10.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a7c151286a7e1d8-0062d4d0b6-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:17:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:17:10.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:10.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:10.140 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:10.192 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:10.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:10.237 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:17:10.260 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:17:10.261 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:17:10.261 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:10.261 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:17:10.262 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:17:10.262 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:17:10.262 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:17:10.263 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:17:10.263 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:17:10.263 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:17:10.263 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:10.264 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:17:09.995026Z" 2022-07-18T03:17:10.264 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:17:10.264 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:17:10.265 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:17:10.265 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:17:10.265 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:17:10.265 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:17:10.267 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:10.267 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:17:10.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:10.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:10.269 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:10.288 INFO:tasks.rgw.c1.client.1.smithi133.stdout:2022-07-18T03:17:10.287+0000 7f94af3ed700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-07-18T03:17:10.345 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:10.345 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:10.472 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.914415_94.1", 2022-07-18T03:17:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:10.484 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.914415Z", 2022-07-18T03:17:10.486 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:10.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:10.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:10.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:10.488 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:10.489 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:10.489 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:10.490 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:10.490 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:10.491 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:10.491 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:10.492 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:10.492 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:10.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:10.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:10.499 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:10.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:10.500 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:10.500 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:10.500 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:10.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:10.501 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:10Z" 2022-07-18T03:17:10.501 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:10.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.914415_94.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.914415Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:10Z" } 2022-07-18T03:17:10.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.914415_94.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.914415Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:10.503 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:10.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.914415_94.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:10.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658114229.995026_105.1 target=1_1658114229.914415_94.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:15.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:15.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:15.509 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:15.560 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:15.560 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:15.633 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:15.633 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:15.633 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:15.651 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:15Z" 2022-07-18T03:17:15.651 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:15.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:15Z" } 2022-07-18T03:17:15.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:15.653 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:15.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:15.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:20.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:20.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:20.657 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:20.708 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:20.709 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:20.793 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:20.805 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:20.805 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:20.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:20.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:20.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:20.806 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:20.806 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:20.806 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:20.807 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:20.807 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:20.807 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:20.807 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:20.808 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:20.808 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:20.808 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:20.808 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:20.809 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:20.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:20.809 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:20.810 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:20.810 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:20.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:20.811 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:20Z" 2022-07-18T03:17:20.811 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:20.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:20Z" } 2022-07-18T03:17:20.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:20.813 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:20.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:20.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:25.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:25.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:25.817 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:25.872 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:25.873 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:25.956 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:25.962 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:25.962 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:25.963 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:25.963 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:25.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:25.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:25.964 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:25.964 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:25.968 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:25.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:25.968 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:25.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:25.969 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:25.969 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:25.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:25.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:25.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:25.970 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:25.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:25.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:25.973 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:25.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:25.974 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:25.974 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:25.974 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:25.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:25.975 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:25Z" 2022-07-18T03:17:25.975 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:25.976 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:25Z" } 2022-07-18T03:17:25.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:25.978 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:25.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:25.979 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:30.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:30.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:30.982 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:31.038 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:31.038 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:31.109 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:31.110 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:31.110 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:31.111 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:31.111 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:31.111 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:31.112 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:31.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:31.112 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:31.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:31.113 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:31.113 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:31.113 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:31.114 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:31.114 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:31.114 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:31.115 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:31.115 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:31.115 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:31.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:31.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:31.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:31.116 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:31.116 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:31.117 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:31.117 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:31.117 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:31.118 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:31.118 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:31.118 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:31.119 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:31.119 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:31.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:31.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:31.120 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:31.120 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:31.121 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:31.121 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:31.121 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:31.121 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:31.122 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:31.122 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:31.122 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:31.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:31.124 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:31.124 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:31.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:31.124 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:31.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:31.125 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:31.125 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:31.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:31.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:31.126 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:31.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:31.126 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:31.126 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:31.127 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:31.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:31.127 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:31Z" 2022-07-18T03:17:31.127 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:31.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:31Z" } 2022-07-18T03:17:31.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:31.130 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:31.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:31.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:36.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:36.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:36.134 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:36.186 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:36.187 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:36.261 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:36.261 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:36.262 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:36.262 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:36.262 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:36.263 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:36.263 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:36.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:36.263 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:36.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:36.264 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:36.264 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:36.264 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:36.265 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:36.265 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:36.265 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:36.265 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:36.266 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:36.266 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:36.266 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:36.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:36.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:36.267 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:36.267 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:36.268 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:36.268 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:36.268 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:36.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:36.269 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:36.270 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:36.270 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:36.270 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:36.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:36.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:36.271 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:36.272 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:36.272 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:36.272 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:36.272 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:36.273 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:36.273 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:36.273 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:36.273 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:36.274 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:36.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:36.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:36.274 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:36.275 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:36.275 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:36.275 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:36.275 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:36.276 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:36.276 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:36.276 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:36.276 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:36.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:36.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:36.277 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:36.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:36.278 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:36.278 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:36.278 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:36.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:36.280 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:36Z" 2022-07-18T03:17:36.280 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:36.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:36Z" } 2022-07-18T03:17:36.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:36.283 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:36.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:36.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:41.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:41.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:41.287 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:41.337 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:41.338 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:41.403 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:41.403 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:41.404 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:41.404 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:41.404 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:41.404 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:41.405 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:41.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:41.405 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:41.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:41.406 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:41.406 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:41.406 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:41.407 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:41.407 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:41.407 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:41.407 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:41.408 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:41.408 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:41.408 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:41.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:41.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:41.409 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:41.409 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:41.409 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:41.410 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:41.410 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:41.410 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:41.410 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:41.411 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:41.411 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:41.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:41.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:41.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:41.412 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:41.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:41.413 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:41.413 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:41.413 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:41.413 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:41.414 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:41.414 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:41.414 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:41.415 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:41.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:41.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:41.415 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:41.416 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:41.416 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:41.416 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:41.416 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:41.417 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:41.417 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:41.417 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:41.417 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:41.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:41.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:41.418 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:41.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:41.419 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:41.419 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:41.419 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:41.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:41.420 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:41Z" 2022-07-18T03:17:41.420 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:41.422 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:41Z" } 2022-07-18T03:17:41.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:41.423 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:41.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:41.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:46.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:46.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:46.428 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:46.489 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:46.490 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:46.559 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:46.560 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:46.560 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:46.561 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:46.561 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:46.566 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:46.566 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:46.566 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:46.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:46.568 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:46.568 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:46.569 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:46.569 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:46.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:46.570 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:46.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:46.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:46.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:46.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:46.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:46.574 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:46.574 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:46.574 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:46.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:46.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:46.575 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:46.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:46.577 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:46.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:46.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:46.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:46.581 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:46.581 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:46.581 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:46.582 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:46.582 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:46.582 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:46.582 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:46.583 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:46.583 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:46.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:46.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:46.584 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:46.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:46.584 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:46.585 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:46.585 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:46.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:46.586 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:46Z" 2022-07-18T03:17:46.586 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:46.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:46Z" } 2022-07-18T03:17:46.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:46.589 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:46.591 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:46.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:51.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:51.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:51.595 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:51.645 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:51.646 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:51.705 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:51.705 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:51.705 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:51.706 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:51.706 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:51.706 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:51.707 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:51.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:51.707 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:51.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:51.708 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:51.708 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:51.709 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:51.709 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:51.710 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:51.711 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:51.711 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:51.724 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:51.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:51.724 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:51Z" 2022-07-18T03:17:51.724 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:51.726 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:51Z" } 2022-07-18T03:17:51.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:51.727 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:51.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:51.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:17:56.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:56.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:17:56.732 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:17:56.784 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:17:56.784 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:17:56.845 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:17:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:17:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:17:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:17:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:17:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:17:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:17:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:17:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:17:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:17:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:17:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:17:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:17:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:17:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:56.858 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:17:56.858 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:56.860 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:56.860 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:17:56.861 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:56.861 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:56.861 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:56.864 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:17:56.864 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:56.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:56.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:56.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:17:56.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:17:56.868 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:17:56.869 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:17:56.869 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:17:56.870 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:17:56.873 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:17:56.873 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:17:56.873 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:17:56.874 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:17:56.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:56.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:17:56.876 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:17:56.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:56.877 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:17:56.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:17:56.878 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:17:56.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:17:56.879 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:17:56Z" 2022-07-18T03:17:56.879 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:17:56.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:17:56Z" } 2022-07-18T03:17:56.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:17:56.883 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:17:56.883 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:17:56.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:01.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:01.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:01.889 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:01.943 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:01.943 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:02.013 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:02.013 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:02.014 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:02.014 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:02.015 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:02.015 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:02.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:02.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:02.017 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:02.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:02.018 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:02.019 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:02.019 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:02.019 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:02.020 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:02.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:02.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:02.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:02.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:02.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:02.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:02.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:02.024 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:02.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:02.025 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:02.025 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:02.025 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:02.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:02.026 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:02.026 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:02.027 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:02.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:02.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:02.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:02.028 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:02.028 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:02.028 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:02.028 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:02.029 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:02.029 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:02.029 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:02.029 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:02.030 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:02.030 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:02.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:02.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:02.031 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:02.031 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:02.031 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:02.031 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:02.032 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:02.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:02.032 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:02.033 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:02.033 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:02.033 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:02.034 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:02.034 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:02.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:02.035 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:02.035 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:02.036 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:02.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:02.036 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:02Z" 2022-07-18T03:18:02.037 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:02.039 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:02Z" } 2022-07-18T03:18:02.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:02.040 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:02.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:02.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:07.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:07.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:07.045 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:07.097 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:07.098 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:07.155 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:07.156 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:07.156 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:07.157 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:07.158 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:07.158 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:07.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:07.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:07.160 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:07.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:07.161 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:07.162 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:07.162 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:07.162 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:07.163 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:07.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:07.164 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:07.164 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:07.165 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:07.165 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:07.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:07.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:07.167 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:07.168 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:07.168 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:07.169 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:07.169 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:07.170 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:07.171 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:07.171 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:07.172 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:07.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:07.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:07.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:07.175 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:07.175 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:07.176 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:07.176 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:07.177 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:07.177 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:07.178 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:07.178 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:07.179 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:07.179 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:07.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:07.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:07.182 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:07.182 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:07.183 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:07.183 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:07.183 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:07.184 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:07.184 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:07.184 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:07.185 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:07.185 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:07.185 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:07.185 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:07.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:07.186 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:07.186 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:07.186 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:07.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:07.187 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:07Z" 2022-07-18T03:18:07.187 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:07.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:07Z" } 2022-07-18T03:18:07.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:07.192 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:07.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:07.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:12.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:12.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:12.196 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:12.251 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:12.252 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:12.333 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:12.333 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:12.333 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:12.334 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:12.334 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:12.334 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:12.335 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:12.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:12.336 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:12.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:12.336 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:12.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:12.337 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:12.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:12.338 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:12.339 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:12.339 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:12.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:12.340 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:12.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:12.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:12.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:12.342 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:12.343 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:12.343 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:12.343 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:12.343 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:12.344 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:12.344 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:12.344 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:12.344 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:12.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:12.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:12.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:12.345 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:12.346 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:12.346 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:12.346 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:12.346 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:12.347 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:12.347 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:12.347 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:12.347 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:12.348 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:12.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:12.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:12.348 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:12.350 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:12.350 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:12.350 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:12.350 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:12.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:12.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:12.351 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:12.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:12.352 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:12.352 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:12.352 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:12.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:12.353 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:12Z" 2022-07-18T03:18:12.353 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:12.354 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:12Z" } 2022-07-18T03:18:12.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:12.355 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:12.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:12.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:17.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:17.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:17.360 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:17.410 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:17.411 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:17.498 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:17.499 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:17.500 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:17.500 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:17.501 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:17.501 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:17.501 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:17.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:17.502 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:17.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:17.503 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:17.503 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:17.504 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:17.504 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:17.505 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:17.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:17.506 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:17.506 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:17.507 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:17.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:17.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:17.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:17.509 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:17.509 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:17.510 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:17.510 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:17.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:17.511 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:17.511 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:17.512 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:17.512 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:17.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:17.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:17.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:17.515 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:17.515 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:17.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:17.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:17.517 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:17.518 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:17.518 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:17.519 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:17.520 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:17.520 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:17.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:17.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:17.522 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:17.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:17.523 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:17.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:17.524 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:17.525 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:17.526 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:17.526 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:17.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:17.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:17.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:17.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:17.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:17.529 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:17.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:17.531 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:17.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:17.532 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:17Z" 2022-07-18T03:18:17.532 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:17.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:17Z" } 2022-07-18T03:18:17.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:17.538 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:17.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:17.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:22.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:22.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:22.544 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:22.602 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:22.603 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:22.693 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:22.693 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:22.693 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:22.694 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:22.694 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:22.694 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:22.694 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:22.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:22.695 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:22.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:22.696 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:22.696 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:22.697 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:22.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:22.698 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:22.698 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:22.698 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:22.699 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:22.699 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:22.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:22.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:22.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:22.701 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:22.701 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:22.702 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:22.703 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:22.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:22.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:22.704 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:22.705 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:22.705 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:22.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:22.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:22.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:22.708 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:22.709 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:22.709 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:22.710 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:22.710 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:22.711 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:22.711 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:22.712 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:22.712 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:22.713 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:22.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:22.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:22.715 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:22.715 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:22.716 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:22.716 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:22.717 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:22.717 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:22.725 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:22Z" 2022-07-18T03:18:22.732 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:22.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:22Z" } 2022-07-18T03:18:22.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:22.736 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:22.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:22.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:27.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:27.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:27.742 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:27.807 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:27.808 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:27.874 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:27.874 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:27.875 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:27.875 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:27.876 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:27.876 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:27.877 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:27.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:27.878 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:27.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:27.878 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:27.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:27.879 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:27.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:27.881 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:27.881 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:27.882 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:27.883 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:27.883 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:27.884 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:27.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:27.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:27.885 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:27.887 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:27.887 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:27.887 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:27.888 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:27.889 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:27.889 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:27.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:27.890 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:27.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:27.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:27.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:27.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:27.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:27.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:27.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:27.893 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:27.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:27.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:27.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:27.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:27.896 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:27.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:27.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:27.897 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:27.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:27.898 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:27.898 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:27.899 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:27.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:27.900 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:27.900 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:27.901 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:27.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:27.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:27.903 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:27.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:27.903 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:27.904 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:27.904 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:27.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:27.904 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:27Z" 2022-07-18T03:18:27.905 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:27.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:27Z" } 2022-07-18T03:18:27.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:27.908 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:27.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:27.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:32.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:32.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:32.912 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:32.961 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:32.962 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:33.037 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:33.037 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:33.038 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:33.039 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:33.039 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:33.040 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:33.040 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:33.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:33.041 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:33.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:33.041 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:33.042 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:33.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:33.043 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:33.043 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:33.043 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:33.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:33.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:33.044 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:33.044 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:33.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:33.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:33.045 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:33.045 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:33.046 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:33.046 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:33.046 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:33.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:33.047 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:33.047 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:33.047 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:33.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:33.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:33.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:33.048 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:33.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:33.049 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:33.049 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:33.049 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:33.049 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:33.050 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:33.050 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:33.050 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:33.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:33.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:33.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:33.051 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:33.051 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:33.052 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:33.052 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:33.053 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:33.053 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:33.054 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:33.054 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:33.056 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:33.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:33.057 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:33.057 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:33.057 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:33.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:33.058 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:33Z" 2022-07-18T03:18:33.058 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:33.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:33Z" } 2022-07-18T03:18:33.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:33.060 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:33.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:33.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:38.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:38.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:38.065 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:38.121 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:38.121 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:38.182 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:38.183 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:38.186 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:38.187 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:38.187 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:38.188 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:38.188 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:38.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:38.189 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:38.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:38.190 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:38.190 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:38.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:38.191 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:38.191 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:38.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:38.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:38.192 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:38.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:38.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:38.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:38.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:38.194 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:38.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:38.194 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:38.194 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:38.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:38.195 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:38.195 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:38.195 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:38.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:38.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:38.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:38.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:38.197 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:38.197 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:38.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:38.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:38.198 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:38.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:38.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:38.199 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:38.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:38.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:38.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:38.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:38.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:38.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:38.200 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:38.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:38.201 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:38.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:38.201 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:38.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:38.202 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:38.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:38.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:38.203 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:38.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:38.203 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:38.203 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:38.204 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:38.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:38.204 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:38Z" 2022-07-18T03:18:38.204 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:38.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:38Z" } 2022-07-18T03:18:38.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:38.207 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:38.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:38.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:43.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:43.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:43.211 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:43.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:43.262 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:43.320 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:43.320 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:43.321 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:43.321 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:43.322 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:43.323 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:43.323 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:43.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:43.324 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:43.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:43.325 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:43.326 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:43.326 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:43.327 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:43.327 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:43.328 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:43.329 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:43.329 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:43.330 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:43.330 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:43.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:43.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:43.332 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:43.332 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:43.333 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:43.333 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:43.334 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:43.334 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:43.335 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:43.335 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:43.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:43.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:43.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:43.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:43.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:43.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:43.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:43.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:43.342 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:43.343 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:43.344 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:43.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:43.345 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:43.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:43.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:43.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:43.346 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:43.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:43.348 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:43.348 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:43.349 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:43.349 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:43.349 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:43.349 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:43.350 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:43.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:43.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:43.351 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:43.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:43.352 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:43.353 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:43.353 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:43.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:43.354 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:43Z" 2022-07-18T03:18:43.354 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:43.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:43Z" } 2022-07-18T03:18:43.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:43.359 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:43.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:43.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:48.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:48.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:48.364 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:48.415 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:48.416 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:48.489 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:48.499 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:48.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:48Z" 2022-07-18T03:18:48.535 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:48.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:48Z" } 2022-07-18T03:18:48.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:48.538 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:48.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:48.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:53.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:53.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:53.543 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:53.592 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:53.593 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:53.663 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:53.663 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:53.664 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:53.664 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:53.664 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:53.665 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:53.665 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:53.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:53.665 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:53.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:53.666 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:53.666 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:53.667 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:53.667 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:53.667 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:53.668 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:53.669 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:53.669 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:53.669 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:53.669 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:53.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:53.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:53.670 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:53.671 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:53.671 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:53.671 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:53.671 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:53Z" 2022-07-18T03:18:53.681 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:53.683 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:53Z" } 2022-07-18T03:18:53.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:53.684 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:53.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:53.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:18:58.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:58.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:18:58.693 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:18:58.743 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:18:58.743 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:18:58.820 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:18:58.821 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:18:58.821 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:18:58.822 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:18:58.822 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:18:58.823 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:18:58.823 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:58.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:58.824 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:18:58.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:58.825 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:18:58.826 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:58.827 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:58.827 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:18:58.828 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:58.828 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:18:58.829 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:58.829 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:18:58.830 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:58.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:58.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:58.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:58.832 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:18:58.832 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:58.833 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:58.833 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:18:58.834 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:58.834 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:58.835 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:58.835 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:18:58.836 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:58.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:58.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:58.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:58.838 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:18:58.838 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:58.838 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:58.838 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:18:58.839 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:58.839 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:58.839 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:58.842 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:18:58.842 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:58.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:58.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:58.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:18:58.850 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:18:58.850 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:18:58.850 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:18:58.851 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:18:58.851 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:18:58.851 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:18:58.851 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:18:58.852 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:18:58.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:18:58.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:58.853 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:18:58.853 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:18:58.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:58.854 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:18:58.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:18:58.854 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:18:58.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:18:58.856 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:18:58Z" 2022-07-18T03:18:58.856 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:18:58.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:18:58Z" } 2022-07-18T03:18:58.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:18:58.861 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:18:58.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:18:58.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:03.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:03.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:03.866 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:03.924 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:03.925 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:04.003 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:04.005 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:04.005 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:04.006 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:04.006 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:04.006 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:04.006 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:04.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:04.007 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:04.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:04.007 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:04.008 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:04.008 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:04.008 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:04.008 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:04.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:04.009 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:04.009 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:04.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:04.010 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:04.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:04.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:04.010 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:04.011 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:04.011 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:04.011 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:04.012 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:04.012 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:04.012 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:04.012 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:04.013 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:04.013 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:04.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:04.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:04.014 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:04.014 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:04.015 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:04.015 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:04.015 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:04.016 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:04.016 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:04.016 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:04.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:04.017 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:04.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:04.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:04.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:04.018 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:04.018 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:04.018 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:04.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:04.019 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:04.019 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:04.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:04.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:04.020 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:04.020 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:04.020 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:04.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:04.021 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:04.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:04.021 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:04.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:04.022 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:03Z" 2022-07-18T03:19:04.022 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:04.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:03Z" } 2022-07-18T03:19:04.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:04.028 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:04.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:04.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:09.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:09.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:09.031 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:09.086 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:09.087 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:09.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:09Z" 2022-07-18T03:19:09.164 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:09.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:09Z" } 2022-07-18T03:19:09.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:09.167 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:09.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:09.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:14.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:14.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:14.173 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:14.229 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:14.230 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:14.319 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:14.320 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:14.320 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:14.321 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:14.321 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:14.322 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:14.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:14.323 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:14.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:14.323 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:14.323 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:14.324 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:14.324 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:14.324 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:14.325 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:14.325 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:14.326 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:14.326 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:14.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:14.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:14.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:14.328 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:14.328 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:14.329 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:14.329 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:14.330 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:14.330 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:14.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:14.331 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:14.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:14.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:14.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:14.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:14.333 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:14.334 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:14.334 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:14.334 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:14.335 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:14.335 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:14.335 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:14.336 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:14.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:14.336 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:14.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:14.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:14.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:14.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:14.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:14.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:14.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:14.339 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:14.340 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:14.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:14.340 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:14.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:14.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:14.341 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:14.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:14.342 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:14.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:14.343 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:14.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:14.344 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:14Z" 2022-07-18T03:19:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:14.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:14Z" } 2022-07-18T03:19:14.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:14.347 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:14.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:14.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:19.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:19.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:19.351 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:19.400 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:19.401 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:19.467 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:19.468 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:19.468 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:19.469 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:19.469 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:19.470 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:19.470 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:19.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:19.471 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:19.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:19.471 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:19.471 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:19.472 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:19.472 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:19.472 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:19.472 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:19.473 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:19.473 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:19.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:19.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:19.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:19.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:19.474 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:19.475 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:19.475 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:19.475 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:19.475 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:19.475 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:19.476 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:19.476 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:19.476 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:19.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:19.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:19.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:19.477 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:19.478 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:19.478 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:19.478 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:19.479 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:19.479 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:19.479 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:19.479 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:19.480 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:19.480 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:19.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:19.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:19.481 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:19.481 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:19.482 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:19.482 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:19.482 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:19.483 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:19.483 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:19.483 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:19.484 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:19.484 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:19.484 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:19.484 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:19.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:19.485 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:19.485 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:19.485 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:19.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:19.486 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:19Z" 2022-07-18T03:19:19.486 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:19.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:19Z" } 2022-07-18T03:19:19.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:19.489 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:19.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:19.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:24.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:24.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:24.493 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:24.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:24.543 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:24.621 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:24.628 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:24.628 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:24.628 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:24.628 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:24.629 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:24.629 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:24.629 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:24.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:24.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:24.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:24.630 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:24.630 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:24.631 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:24.631 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:24.631 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:24.631 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:24.632 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:24.632 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:24.632 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:24.632 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:24.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:24.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:24.633 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:24.633 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:24.634 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:24.634 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:24.634 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:24.634 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:24.635 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:24.635 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:24.635 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:24.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:24.636 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:24.636 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:24.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:24.636 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:24.637 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:24.637 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:24.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:24.637 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:24Z" 2022-07-18T03:19:24.638 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:24.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:24Z" } 2022-07-18T03:19:24.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:24.640 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:24.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:24.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:29.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:29.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:29.644 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:29.693 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:29.694 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:29.771 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:29Z" 2022-07-18T03:19:29.804 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:29.807 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:29Z" } 2022-07-18T03:19:29.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:29.809 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:29.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:29.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:34.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:34.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:34.814 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:34.870 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:34.870 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:34.946 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:34.947 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:34.947 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:34.947 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:34.948 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:34.948 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:34.948 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:34.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:34.949 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:34.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:34.949 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:34.949 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:34.950 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:34.950 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:34.950 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:34.950 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:34.951 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:34.951 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:34.952 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:34.952 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:34.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:34.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:34.953 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:34.953 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:34.953 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:34.953 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:34.954 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:34.954 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:34.954 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:34.954 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:34.955 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:34.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:34.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:34.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:34.956 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:34.956 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:34.958 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:34.958 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:34.958 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:34.959 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:34.959 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:34.959 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:34.960 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:34.960 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:34.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:34.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:34.962 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:34.962 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:34.963 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:34.963 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:34.963 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:34.964 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:34.964 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:34.964 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:34.964 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:34.965 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:34.965 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:34.965 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:34.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:34.966 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:34.966 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:34.967 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:34.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:34.967 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:34Z" 2022-07-18T03:19:34.968 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:34.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:34Z" } 2022-07-18T03:19:34.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:34.971 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:34.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:34.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:39.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:39.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:39.975 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:40.036 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:40.037 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:40.188 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:40.189 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:40.189 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:40.189 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:40.190 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:40.190 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:40.191 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:40.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:40.191 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:40.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:40.192 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:40.192 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:40.192 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:40.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:40.193 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:40.193 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:40.194 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:40.194 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:40.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:40.195 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:40.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:40.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:40.196 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:40.196 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:40.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:40.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:40.197 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:40.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:40.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:40.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:40.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:40.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:40.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:40.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:40.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:40.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:40.200 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:40.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:40.201 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:40.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:40.202 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:40.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:40.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:40.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:40.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:40.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:40.204 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:40.204 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:40.204 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:40.205 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:40.205 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:40.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:40.206 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:40.206 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:40.206 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:40.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:40.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:40.207 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:40.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:40.208 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:40.208 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:40.209 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:40.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:40.210 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:40Z" 2022-07-18T03:19:40.210 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:40.213 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:40Z" } 2022-07-18T03:19:40.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:40.215 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:40.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:40.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:45.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:45.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:45.221 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:45.276 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:45.277 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:45.351 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:45.352 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:45.352 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:45.352 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:45.353 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:45.353 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:45.353 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:45.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:45.354 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:45.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:45.355 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:45.355 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:45.355 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:45.356 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:45.357 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:45.357 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:45.358 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:45.358 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:45.359 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:45.359 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:45.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:45.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:45.360 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:45.361 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:45.361 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:45.362 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:45.362 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:45.363 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:45.363 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:45.364 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:45.364 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:45.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:45.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:45.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:45.366 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:45.366 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:45.366 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:45.367 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:45.367 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:45.368 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:45.368 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:45.368 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:45.369 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:45.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:45.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:45.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:45.370 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:45.370 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:45.370 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:45.371 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:45.371 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:45.371 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:45.371 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:45.372 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:45.372 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:45.372 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:45.372 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:45.373 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:45.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:45.373 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:45.373 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:45.374 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:45.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:45.374 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:45Z" 2022-07-18T03:19:45.375 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:45.377 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:45Z" } 2022-07-18T03:19:45.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:45.379 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:45.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:45.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:50.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:50.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:50.387 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:50.437 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:50.437 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:50.512 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:50.513 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:50.513 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:50.514 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:50.515 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:50.515 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:50.516 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:50.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:50.517 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:50.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:50.518 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:50.518 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:50.519 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:50.519 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:50.519 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:50.520 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:50.520 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:50.521 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:50.521 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:50.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:50.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:50.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:50.523 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:50.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:50.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:50.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:50.524 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:50.525 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:50.525 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:50.525 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:50.526 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:50.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:50.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:50.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:50.528 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:50.528 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:50.528 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:50.529 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:50.529 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:50.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:50.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:50.530 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:50.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:50.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:50.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:50.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:50.532 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:50.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:50.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:50.534 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:50.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:50.535 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:50.537 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:50.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:50.538 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:50.538 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:50.539 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:50.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:50Z" 2022-07-18T03:19:50.540 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:50.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:50Z" } 2022-07-18T03:19:50.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:50.543 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:50.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:50.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:19:55.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:55.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:19:55.547 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:19:55.602 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:19:55.603 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:19:55.680 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:19:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:19:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:19:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:19:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:19:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:19:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:19:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:19:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:19:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:19:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:19:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:19:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:19:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:19:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:19:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:19:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:19:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:19:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:19:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:19:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:19:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:19:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:19:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:19:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:19:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:19:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:19:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:19:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:19:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:19:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:19:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:19:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:19:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:19:55Z" 2022-07-18T03:19:55.704 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:19:55.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:19:55Z" } 2022-07-18T03:19:55.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:19:55.707 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:19:55.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:19:55.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:00.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:00.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:00.710 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:00.759 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:00.760 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:00.835 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:00.836 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:00.836 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:00.837 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:00.837 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:00.837 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:00.837 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:00.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:00.838 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:00.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:00.838 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:00.839 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:00.839 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:00.839 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:00.839 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:00.840 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:00.840 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:00.840 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:00.840 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:00.841 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:00.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:00.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:00.842 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:00.842 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:00.842 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:00.842 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:00.843 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:00.843 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:00.843 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:00.844 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:00.844 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:00.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:00.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:00.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:00.845 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:00.845 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:00.845 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:00.846 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:00.846 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:00.846 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:00.847 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:00.847 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:00.848 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:00.848 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:00.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:00.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:00.849 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:00.849 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:00.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:00.852 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:00.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:00.852 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:00.853 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:00.853 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:00.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:00.853 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:00Z" 2022-07-18T03:20:00.854 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:00.855 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:00Z" } 2022-07-18T03:20:00.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:00.856 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:00.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:00.857 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:05.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:05.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:05.860 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:05.915 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:05.916 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:05.991 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:05Z" 2022-07-18T03:20:06.017 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:06.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:05Z" } 2022-07-18T03:20:06.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:06.019 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:06.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:06.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:11.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:11.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:11.024 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:11.073 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:11.073 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:11.132 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:11.133 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:11.133 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:11.134 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:11.134 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:11.135 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:11.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:11.136 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:11.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:11.137 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:11.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:11.138 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:11.138 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:11.139 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:11.139 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:11.140 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:11.141 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:11.142 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:11.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:11.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:11.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:11.144 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:11.144 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:11.145 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:11.145 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:11.145 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:11.146 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:11.146 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:11.147 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:11.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:11.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:11.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:11.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:11.149 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:11.150 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:11.150 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:11.150 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:11.150 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:11.151 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:11.151 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:11.151 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:11.151 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:11.152 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:11.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:11.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:11.152 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:11.153 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:11.153 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:11.153 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:11.153 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:11.154 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:11.154 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:11.154 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:11.154 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:11.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:11.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:11.155 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:11.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:11.156 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:11.156 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:11.156 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:11.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:11.157 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:11Z" 2022-07-18T03:20:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:11.158 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:11Z" } 2022-07-18T03:20:11.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:11.160 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:11.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:11.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:16.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:16.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:16.164 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:16.212 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:16.212 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:16.282 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:16.283 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:16.284 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:16.284 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:16.285 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:16.285 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:16.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:16.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:16.286 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:16.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:16.288 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:16.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:16.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:16.289 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:16.289 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:16.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:16.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:16.290 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:16.290 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:16.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:16.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:16.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:16.291 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:16.291 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:16.292 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:16.292 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:16.293 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:16.293 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:16.293 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:16.294 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:16.294 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:16.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:16.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:16.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:16.295 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:16.295 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:16.295 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:16.296 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:16.296 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:16.296 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:16.297 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:16.297 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:16.297 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:16.297 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:16.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:16.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:16.298 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:16.299 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:16.299 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:16.299 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:16.299 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:16.300 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:16.300 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:16.300 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:16.300 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:16.301 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:16.301 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:16.301 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:16.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:16.302 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:16.303 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:16.303 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:16.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:16.304 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:16Z" 2022-07-18T03:20:16.304 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:16.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:16Z" } 2022-07-18T03:20:16.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:16.307 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:16.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:16.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:21.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:21.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:21.311 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:21.379 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:21.379 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:21.524 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:21.526 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:21.526 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:21.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:21.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:21.529 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:21.529 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:21.529 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:21.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:21.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:21.530 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:21.530 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:21.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:21.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:21.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:21.531 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:21.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:21.532 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:21.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:21.534 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:21.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:21.535 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:21.535 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:21.536 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:21.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:21.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:21.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:21.537 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:21.537 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:21.537 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:21.538 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:21.538 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:21.538 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:21.538 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:21.539 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:21.539 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:21.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:21.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:21.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:21.540 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:21.540 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:21.540 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:21.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:21.541 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:21.541 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:21.541 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:21.542 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:21.542 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:21.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:21.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:21.543 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:21.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:21.543 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:21.543 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:21.544 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:21.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:21.544 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:21Z" 2022-07-18T03:20:21.544 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:21.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:21Z" } 2022-07-18T03:20:21.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:21.547 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:21.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:21.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:26.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:26.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:26.551 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:26.606 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:26.607 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:26.673 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:26.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:26.677 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:26.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:26.678 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:26.679 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:26.679 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:26.692 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:26.692 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:26.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:26.692 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:26Z" 2022-07-18T03:20:26.693 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:26.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:26Z" } 2022-07-18T03:20:26.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:26.695 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:26.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:26.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:31.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:31.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:31.699 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:31.755 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:31.756 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:31.848 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:31.851 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:31.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:31.851 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:31.851 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:31.852 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:31.852 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:31.852 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:31.852 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:31.853 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:31.853 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:31.853 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:31.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:31.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:31.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:31.854 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:31.855 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:31.855 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:31.855 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:31.855 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:31.856 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:31.856 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:31.856 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:31.856 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:31.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:31.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:31.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:31.858 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:31.858 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:31.858 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:31.858 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:31.859 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:31.859 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:31.859 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:31.859 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:31.860 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:31.860 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:31.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:31.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:31.861 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:31.861 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:31.861 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:31.861 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:31.862 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:31.862 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:31.862 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:31.862 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:31.863 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:31.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:31.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:31.863 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:31.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:31.864 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:31.864 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:31.864 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:31.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:31.865 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:31Z" 2022-07-18T03:20:31.865 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:31.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:31Z" } 2022-07-18T03:20:31.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:31.868 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:31.868 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:31.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:36.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:36.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:36.872 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:36.931 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:36.931 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:36.993 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:36.994 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:36.994 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:36.995 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:36.995 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:36.996 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:36.996 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:36.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:36.997 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:36.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:36.998 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:36.998 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:36.999 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:36.999 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:37.001 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:37.002 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:37.002 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:37.003 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:37.003 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:37.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:37.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:37.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:37.005 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:37.006 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:37.007 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:37.007 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:37.007 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:37.008 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:37.008 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:37.008 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:37.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:37.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:37.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:37.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:37.011 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:37.012 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:37.012 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:37.013 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:37.013 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:37.014 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:37.014 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:37.015 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:37.015 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:37.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:37.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:37.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:37.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:37.018 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:37.018 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:37.019 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:37.019 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:37.020 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:37.021 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:37.021 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:37.021 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:37.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:37.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:37.023 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:37.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:37.023 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:37.024 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:37.024 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:37.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:37.025 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:36Z" 2022-07-18T03:20:37.025 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:37.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:36Z" } 2022-07-18T03:20:37.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:37.028 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:37.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:37.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:42.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:42.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:42.032 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:42.082 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:42.083 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:42.157 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:42.157 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:42.158 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:42.158 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:42.158 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:42.159 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:42.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:42.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:42.160 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:42.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:42.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:42.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:42.161 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:42.161 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:42.161 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:42.161 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:42.162 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:42.162 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:42.162 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:42.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:42.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:42.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:42.163 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:42.164 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:42.164 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:42.164 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:42.164 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:42.165 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:42.165 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:42.165 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:42.165 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:42.166 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:42.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:42.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:42.166 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:42.167 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:42.167 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:42.167 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:42.167 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:42.168 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:42.168 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:42.168 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:42.168 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:42.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:42.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:42.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:42.170 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:42.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:42.170 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:42.170 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:42.171 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:42.171 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:42.171 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:42.171 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:42.172 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:42.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:42.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:42.172 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:42.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:42.173 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:42.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:42.173 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:42.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:42.174 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:42Z" 2022-07-18T03:20:42.174 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:42.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:42Z" } 2022-07-18T03:20:42.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:42.177 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:42.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:42.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:47.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:47.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:47.181 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:47.237 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:47.237 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:47.307 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:47.308 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:47.308 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:47.309 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:47.310 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:47.310 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:47.311 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:47.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:47.312 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:47.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:47.313 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:47.313 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:47.314 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:47.314 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:47.314 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:47.314 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:47.315 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:47.315 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:47.315 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:47.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:47.316 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:47.317 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:47.318 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:47.319 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:47.320 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:47.320 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:47.320 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:47.320 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:47.321 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:47.321 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:47.321 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:47.321 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:47.322 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:47.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:47.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:47.322 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:47.323 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:47.323 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:47.323 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:47.324 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:47.324 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:47.324 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:47.324 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:47.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:47.325 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:47.325 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:47.325 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:47.326 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:47.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:47.327 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:47Z" 2022-07-18T03:20:47.327 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:47.329 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:47Z" } 2022-07-18T03:20:47.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:47.330 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:47.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:47.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:52.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:52.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:52.334 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:52.383 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:52.383 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:52.446 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:52.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:52.449 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:52.454 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:52.454 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:52.454 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:52.454 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:52.455 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:52.455 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:52.455 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:52.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:52.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:52.456 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:52.457 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:52.457 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:52.457 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:52.457 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:52.458 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:52.458 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:52.458 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:52.458 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:52.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:52.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:52.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:52.459 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:52.460 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:52.460 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:52.460 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:52.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:52.461 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:52.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:52.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:52.462 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:52.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:52.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:52.463 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:52.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:52.463 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:52.464 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:52.464 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:52.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:52.465 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:52Z" 2022-07-18T03:20:52.465 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:52.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:52Z" } 2022-07-18T03:20:52.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:52.468 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:52.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:52.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:20:57.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:57.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:20:57.473 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:20:57.523 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:20:57.524 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:20:57.577 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:20:57.577 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:20:57.578 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:20:57.578 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:20:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:20:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:20:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:57.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:57.580 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:20:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:20:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:57.582 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:57.582 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:20:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:20:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:20:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:20:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:20:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:20:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:20:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:20:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:20:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:20:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:20:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:20:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:20:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:20:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:20:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:20:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:20:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:20:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:20:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:20:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:20:57.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:57.594 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:20:57.594 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:20:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:20:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:20:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:20:57Z" 2022-07-18T03:20:57.595 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:20:57.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:20:57Z" } 2022-07-18T03:20:57.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:20:57.598 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:20:57.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:20:57.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:02.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:02.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:02.603 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:02.659 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:02.659 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:02.720 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:02.724 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:02.724 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:02.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:02Z" 2022-07-18T03:21:02.738 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:02.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:02Z" } 2022-07-18T03:21:02.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:02.741 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:02.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:02.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:07.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:07.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:07.745 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:07.800 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:07.800 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:07.874 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:07.874 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:07.875 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:07.875 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:07.876 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:07.876 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:07.877 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:07.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:07.878 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:07.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:07.878 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:07.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:07.880 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:07.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:07.881 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:07.881 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:07.882 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:07.882 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:07.883 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:07.901 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:07.901 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:07Z" 2022-07-18T03:21:07.903 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:07.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:07Z" } 2022-07-18T03:21:07.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:07.906 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:07.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:07.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:12.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:12.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:12.911 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:12.962 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:12.962 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:13.017 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:13.017 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:13.018 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:13.018 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:13.018 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:13.019 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:13.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:13.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:13.020 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:13.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:13.021 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:13.021 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:13.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:13.022 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:13.022 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:13.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:13.026 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:13.026 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:13.027 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:13.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:13.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:13.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:13.029 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:13.029 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:13.030 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:13.031 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:13.031 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:13.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:13.032 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:13.033 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:13.039 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:13.040 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:13.040 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:13.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:13.043 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:13.043 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:13Z" 2022-07-18T03:21:13.044 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:13.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:13Z" } 2022-07-18T03:21:13.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:13.047 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:13.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:13.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:18.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:18.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:18.052 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:18.110 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:18.111 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:18.182 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:18.183 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:18.183 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:18.183 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:18.184 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:18.184 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:18.184 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:18.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:18.185 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:18.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:18.185 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:18.185 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:18.186 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:18.186 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:18.186 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:18.186 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:18.187 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:18.187 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:18.187 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:18.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:18.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:18.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:18.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:18.203 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:18.203 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:18.204 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:18.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:18.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:18.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:18.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:18.206 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:18.206 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:18.207 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:18.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:18.208 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:18Z" 2022-07-18T03:21:18.208 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:18.210 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:18Z" } 2022-07-18T03:21:18.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:18.212 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:18.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:18.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:23.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:23.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:23.217 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:23.267 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:23.268 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:23.348 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:23.349 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:23.349 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:23.350 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:23.350 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:23.354 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:23.354 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:23.356 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:23.356 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:23.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:23.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:23.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:23.359 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:23.359 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:23.360 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:23.361 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:23.362 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:23.362 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:23Z" 2022-07-18T03:21:23.383 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:23.386 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:23Z" } 2022-07-18T03:21:23.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:23.389 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:23.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:23.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:28.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:28.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:28.394 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:28.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:28.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:28.524 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:28.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:28.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:28.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:28.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:28.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:28.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:28.536 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:28.536 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:28.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:28.537 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:28.537 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:28.537 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:28.538 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:28.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:28.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:28.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:28.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:28.539 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:28.539 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:28.541 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:28.541 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:28.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:28.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:28.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:28.542 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:28.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:28.543 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:28.543 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:28.543 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:28.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:28.544 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:28Z" 2022-07-18T03:21:28.544 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:28.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:28Z" } 2022-07-18T03:21:28.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:28.547 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:28.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:28.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:33.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:33.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:33.555 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:33.604 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:33.605 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:33.658 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:33.659 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:33.659 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:33.660 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:33.660 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:33.661 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:33.661 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:33.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:33.663 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:33.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:33.664 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:33.664 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:33.665 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:33.665 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:33.666 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:33.666 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:33.667 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:33.667 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:33.668 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:33.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:33.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:33.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:33.670 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:33.670 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:33.671 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:33.671 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:33.672 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:33.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:33.673 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:33.673 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:33.674 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:33.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:33.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:33.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:33.675 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:33.676 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:33.676 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:33.677 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:33.677 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:33.678 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:33.678 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:33.679 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:33.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:33.680 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:33.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:33.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:33.681 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:33.682 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:33.682 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:33.683 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:33.683 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:33.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:33.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:33Z" 2022-07-18T03:21:33.690 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:33.692 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:33Z" } 2022-07-18T03:21:33.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:33.694 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:33.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:33.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:38.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:38.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:38.699 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:38.756 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:38.756 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:38.822 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:38.824 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:38.824 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:38.824 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:38.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:38.825 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:38.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:38.826 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:38.839 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:38.840 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:38.840 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:38.841 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:38.841 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:38.841 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:38.842 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:38.842 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:38.842 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:38.842 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:38.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:38.843 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:38.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:38.843 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:38.844 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:38.844 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:38.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:38.845 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:38Z" 2022-07-18T03:21:38.845 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:38.846 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:38Z" } 2022-07-18T03:21:38.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:38.848 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:38.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:38.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:43.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:43.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:43.853 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:43.910 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:43.910 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:43.992 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:43.993 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:43.993 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:43.993 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:43.994 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:43.994 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:43.994 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:43.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:43.995 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:43.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:43.995 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:43.997 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:43.997 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:43.997 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:43.997 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:43.998 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:43.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:43.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:43.998 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:44.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:44.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:44.005 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:44.006 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:44.006 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:44.006 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:44.006 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:44.007 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:44.007 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:44.007 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:44.008 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:44.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:44.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:44.008 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:44.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:44.009 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:44.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:44.009 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:44.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:44.010 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:43Z" 2022-07-18T03:21:44.010 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:44.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:43Z" } 2022-07-18T03:21:44.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:44.013 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:44.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:44.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:49.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:49.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:49.019 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:49.073 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:49.074 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:49.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:49Z" 2022-07-18T03:21:49.166 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:49.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:49Z" } 2022-07-18T03:21:49.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:49.168 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:49.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:49.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:54.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:54.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:54.172 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:54.228 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:54.229 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:54.303 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:54.303 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:54.304 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:54.305 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:54.306 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:54.306 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:54.307 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:54.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:54.308 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:54.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:54.309 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:54.310 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:54.310 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:54.311 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:54.311 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:54.312 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:54.312 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:54.313 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:54.313 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:54.314 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:54.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:54.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:54.316 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:54.317 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:54.317 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:54.318 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:54.318 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:54.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:54.319 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:54.320 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:54.320 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:54.321 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:54.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:54.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:54.322 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:54.323 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:54.323 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:54.325 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:54.325 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:54.326 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:54.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:54.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:54.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:54.328 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:54.328 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:54.329 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:54.329 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:54.330 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:54.330 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:54.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:54.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:54.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:54.333 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:54.333 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:54.334 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:54.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:54.336 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:54Z" 2022-07-18T03:21:54.336 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:54.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:54Z" } 2022-07-18T03:21:54.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:54.339 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:54.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:54.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:21:59.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:59.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:21:59.344 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:21:59.394 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:21:59.395 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:21:59.452 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:21:59.453 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:21:59.453 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:21:59.453 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:21:59.453 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:21:59.454 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:21:59.455 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:59.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:59.456 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:21:59.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:59.456 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:21:59.457 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:59.457 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:59.457 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:21:59.457 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:59.458 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:21:59.458 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:59.458 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:21:59.459 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:59.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:59.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:59.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:59.460 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:21:59.460 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:59.460 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:59.460 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:21:59.461 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:59.461 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:59.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:59.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:21:59.462 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:59.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:59.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:59.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:59.463 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:21:59.463 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:59.463 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:59.464 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:21:59.464 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:59.464 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:59.464 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:59.465 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:21:59.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:59.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:59.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:59.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:21:59.466 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:21:59.466 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:21:59.467 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:21:59.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:21:59.467 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:21:59.467 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:21:59.468 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:21:59.468 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:21:59.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:21:59.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:21:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:21:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:21:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:21:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:21:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:21:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:21:59Z" 2022-07-18T03:21:59.471 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:21:59.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:21:59Z" } 2022-07-18T03:21:59.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:21:59.475 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:21:59.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:21:59.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:04.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:04.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:04.480 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:04.535 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:04.536 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:04.615 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:04.616 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:04.616 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:04.617 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:04.617 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:04.617 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:04.618 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:04.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:04.619 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:04.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:04.619 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:04.620 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:04.620 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:04.621 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:04.621 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:04.622 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:04.622 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:04.622 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:04.622 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:04.623 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:04.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:04.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:04.623 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:04.624 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:04.624 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:04.624 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:04.624 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:04.625 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:04.625 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:04.625 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:04.625 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:04.626 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:04.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:04.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:04.626 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:04.627 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:04.627 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:04.627 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:22:04.627 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:04.628 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:04.628 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:04.628 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:22:04.628 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:04.630 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:04.630 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:04.630 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:04.630 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:04.631 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:04.631 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:04.631 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:04.631 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:04Z" 2022-07-18T03:22:04.634 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:04.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:04Z" } 2022-07-18T03:22:04.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:04.635 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:04.636 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:04.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:09.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:09.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:09.639 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:09.691 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:09.692 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:09.762 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:09.776 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:22:09.776 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:09.776 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:09.776 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:09.777 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:22:09.777 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:09.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:09.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:09.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:09.778 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:09.778 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:09.778 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:09.779 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:09.779 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:09.779 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:09.780 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:09.780 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:09.781 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:09.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:09.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:09.782 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:09.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:09.783 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:09.783 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:09.783 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:09.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:09.784 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:09Z" 2022-07-18T03:22:09.784 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:09.785 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:09Z" } 2022-07-18T03:22:09.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:09.786 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:09.786 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:09.787 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:14.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:14.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:14.790 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:14.864 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:14.864 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:14.944 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:14.945 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:14.945 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:14.946 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:14.946 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:14.947 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:14.947 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:14.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:14.949 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:14.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:14.951 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:14.951 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:14.952 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:14.952 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:14.953 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:14.953 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:14.954 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:14.955 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:14.955 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:14.956 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:14.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:14.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:14.957 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:14.958 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:14.959 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:14.960 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:14.961 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:14.961 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:14.962 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:14.962 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:14.963 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:14.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:14.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:14.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:14.965 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:14.966 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:14.967 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:14.967 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114229.995026_105.1", 2022-07-18T03:22:14.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:14.969 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:14.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:14.970 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:17:09.995026Z", 2022-07-18T03:22:14.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:14.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:14.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:14.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:14Z" 2022-07-18T03:22:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:14.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114229.995026_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:17:09.995026Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:14Z" } 2022-07-18T03:22:14.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114229.995026_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:17:09.995026Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:14.986 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:14.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114229.995026_105.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:14.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:19.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... FAIL 2022-07-18T03:22:19.998 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-16 2022-07-18T03:22:19.999 DEBUG:boto:path=/mlvank-16/ 2022-07-18T03:22:19.999 DEBUG:boto:auth_path=/mlvank-16/ 2022-07-18T03:22:20.000 DEBUG:boto:Method: PUT 2022-07-18T03:22:20.001 DEBUG:boto:Path: /mlvank-16/ 2022-07-18T03:22:20.002 DEBUG:boto:Data: 2022-07-18T03:22:20.002 DEBUG:boto:Headers: {} 2022-07-18T03:22:20.003 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:22:20.004 DEBUG:boto:Port: 8000 2022-07-18T03:22:20.005 DEBUG:boto:Params: {} 2022-07-18T03:22:20.006 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:22:20.007 DEBUG:boto:Token: None 2022-07-18T03:22:20.008 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:22:20 GMT /mlvank-16/ 2022-07-18T03:22:20.009 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:4RAEiijwoksVPisGa6ykCSZ6dT0= 2022-07-18T03:22:20.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:22:20 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:4RAEiijwoksVPisGa6ykCSZ6dT0=', 'Content-Length': '0'} 2022-07-18T03:22:20.038 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f858a5ac266afa3d-0062d4d1ec-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:22:20 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:22:20.040 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-17 2022-07-18T03:22:20.040 DEBUG:boto:path=/mlvank-17/ 2022-07-18T03:22:20.041 DEBUG:boto:auth_path=/mlvank-17/ 2022-07-18T03:22:20.041 DEBUG:boto:Method: PUT 2022-07-18T03:22:20.042 DEBUG:boto:Path: /mlvank-17/ 2022-07-18T03:22:20.042 DEBUG:boto:Data: 2022-07-18T03:22:20.043 DEBUG:boto:Headers: {} 2022-07-18T03:22:20.043 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:22:20.044 DEBUG:boto:Port: 8000 2022-07-18T03:22:20.045 DEBUG:boto:Params: {} 2022-07-18T03:22:20.046 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:22:20.047 DEBUG:boto:Token: None 2022-07-18T03:22:20.049 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:22:20 GMT /mlvank-17/ 2022-07-18T03:22:20.050 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:tXr8cD2dHCt5xAEFCRVPWkI42qU= 2022-07-18T03:22:20.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:22:20 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:tXr8cD2dHCt5xAEFCRVPWkI42qU=', 'Content-Length': '0'} 2022-07-18T03:22:20.087 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ee65c25efbfcefb-0062d4d1ec-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:22:20 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:22:20.087 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-18 2022-07-18T03:22:20.088 DEBUG:boto:path=/mlvank-18/ 2022-07-18T03:22:20.088 DEBUG:boto:auth_path=/mlvank-18/ 2022-07-18T03:22:20.089 DEBUG:boto:Method: PUT 2022-07-18T03:22:20.090 DEBUG:boto:Path: /mlvank-18/ 2022-07-18T03:22:20.090 DEBUG:boto:Data: 2022-07-18T03:22:20.091 DEBUG:boto:Headers: {} 2022-07-18T03:22:20.091 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:22:20.092 DEBUG:boto:Port: 8001 2022-07-18T03:22:20.092 DEBUG:boto:Params: {} 2022-07-18T03:22:20.093 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:22:20.094 DEBUG:boto:Token: None 2022-07-18T03:22:20.094 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:22:20 GMT /mlvank-18/ 2022-07-18T03:22:20.095 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:iZwecGDjPTAPzJgUGNFPJiK2GtI= 2022-07-18T03:22:20.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:22:20 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:iZwecGDjPTAPzJgUGNFPJiK2GtI=', 'Content-Length': '0'} 2022-07-18T03:22:20.128 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000065eac17c96e0d79-0062d4d1ec-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:22:20 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:22:20.130 DEBUG:boto:path=/mlvank-16/ 2022-07-18T03:22:20.135 DEBUG:boto:auth_path=/mlvank-16/ 2022-07-18T03:22:20.136 DEBUG:boto:path=/mlvank-16/?versioning 2022-07-18T03:22:20.136 DEBUG:boto:auth_path=/mlvank-16/?versioning 2022-07-18T03:22:20.137 DEBUG:boto:Method: PUT 2022-07-18T03:22:20.137 DEBUG:boto:Path: /mlvank-16/?versioning 2022-07-18T03:22:20.138 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:22:20.139 DEBUG:boto:Headers: {} 2022-07-18T03:22:20.139 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:22:20.140 DEBUG:boto:Port: 8000 2022-07-18T03:22:20.140 DEBUG:boto:Params: {} 2022-07-18T03:22:20.141 DEBUG:boto:Token: None 2022-07-18T03:22:20.142 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:22:20 GMT /mlvank-16/?versioning 2022-07-18T03:22:20.142 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:zWD38NFeu1YXuWO6cxOCoAxx3CY= 2022-07-18T03:22:20.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:22:20 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:zWD38NFeu1YXuWO6cxOCoAxx3CY=', 'Content-Length': '228'} 2022-07-18T03:22:20.149 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001363d5e5b1509a60-0062d4d1ec-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:22:20 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:22:20.150 DEBUG:boto:path=/mlvank-17/ 2022-07-18T03:22:20.150 DEBUG:boto:auth_path=/mlvank-17/ 2022-07-18T03:22:20.151 DEBUG:boto:path=/mlvank-17/?versioning 2022-07-18T03:22:20.151 DEBUG:boto:auth_path=/mlvank-17/?versioning 2022-07-18T03:22:20.152 DEBUG:boto:Method: PUT 2022-07-18T03:22:20.152 DEBUG:boto:Path: /mlvank-17/?versioning 2022-07-18T03:22:20.153 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:22:20.153 DEBUG:boto:Headers: {} 2022-07-18T03:22:20.154 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:22:20.154 DEBUG:boto:Port: 8000 2022-07-18T03:22:20.155 DEBUG:boto:Params: {} 2022-07-18T03:22:20.156 DEBUG:boto:Token: None 2022-07-18T03:22:20.156 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:22:20 GMT /mlvank-17/?versioning 2022-07-18T03:22:20.157 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:4IuRLqkiuKHshvXDX5T+3tLrCuU= 2022-07-18T03:22:20.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:22:20 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:4IuRLqkiuKHshvXDX5T+3tLrCuU=', 'Content-Length': '228'} 2022-07-18T03:22:20.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006afea77fbd545f58-0062d4d1ec-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:22:20 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:22:20.182 DEBUG:boto:path=/mlvank-18/ 2022-07-18T03:22:20.182 DEBUG:boto:auth_path=/mlvank-18/ 2022-07-18T03:22:20.183 DEBUG:boto:path=/mlvank-18/?versioning 2022-07-18T03:22:20.183 DEBUG:boto:auth_path=/mlvank-18/?versioning 2022-07-18T03:22:20.184 DEBUG:boto:Method: PUT 2022-07-18T03:22:20.185 DEBUG:boto:Path: /mlvank-18/?versioning 2022-07-18T03:22:20.185 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:22:20.186 DEBUG:boto:Headers: {} 2022-07-18T03:22:20.186 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:22:20.187 DEBUG:boto:Port: 8001 2022-07-18T03:22:20.187 DEBUG:boto:Params: {} 2022-07-18T03:22:20.188 DEBUG:boto:Token: None 2022-07-18T03:22:20.189 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:22:20 GMT /mlvank-18/?versioning 2022-07-18T03:22:20.190 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:7lDwEWRADGij2svbc1Qm9AI0j4A= 2022-07-18T03:22:20.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:22:20 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:7lDwEWRADGij2svbc1Qm9AI0j4A=', 'Content-Length': '228'} 2022-07-18T03:22:20.203 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000edc036b515923553-0062d4d1ec-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:22:20 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:22:20.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:20.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:20.206 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:20.262 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:20.263 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:20.324 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:22:20.336 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:22:20.336 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:22:20.337 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:20.337 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:22:20.337 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:22:20.337 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:22:20.338 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:22:20.338 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:22:20.338 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:22:20.339 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:22:20.339 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:20.339 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:22:20.200245Z" 2022-07-18T03:22:20.339 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:22:20.340 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:22:20.340 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:22:20.340 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:22:20.341 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:22:20.341 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:22:20.342 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:20.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:22:20.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:20.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:20.345 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:20.408 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:20.409 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:20.525 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:20.526 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:20.526 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:20.526 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:20.527 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:20.527 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:20.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:20.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:20.528 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:20.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:20.528 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:20.528 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:20.529 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:20.529 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:20.529 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:20.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:20.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:20.530 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:20.530 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:20.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:20.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:20.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:20.531 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:20.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:20.532 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:20.532 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:20.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:20.533 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:20.533 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:20.533 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:20.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:20.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:20.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:20.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:20.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:20.535 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:20.535 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:20.535 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.026122_113.1", 2022-07-18T03:22:20.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:20.537 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:20.538 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:20.538 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.026122Z", 2022-07-18T03:22:20.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:20.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:20.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:20.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:20.539 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:20.540 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:20.540 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:20.540 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:20.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:20.541 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:20.541 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:20.541 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:20.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:20.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:20.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:20.542 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:20.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:20.546 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:20.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:20.547 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:20.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:20.547 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:20Z" 2022-07-18T03:22:20.547 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:20.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.026122_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.026122Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:20Z" } 2022-07-18T03:22:20.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.026122_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.026122Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:20.550 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:20.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.026122_113.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:20.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658114540.200245_130.1 target=1_1658114540.026122_113.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:25.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:25.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:25.555 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:25.607 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:25.608 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:25.689 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:25.690 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:25.690 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:25.690 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:25.690 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:25.691 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:25.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:25.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:25.691 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:25.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:25.692 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:25.692 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:25.693 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:25.693 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:25.693 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:25.693 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:25.694 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:25.694 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:25.694 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:25.694 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:25.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:25.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:25.695 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:25.695 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:25.696 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:25.696 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:25.696 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:25.696 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:25.697 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:25.697 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:25.697 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:25.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:25.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:25.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:25.698 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:25.698 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:25.699 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:25.699 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:25.699 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:25.699 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:25.700 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:25.700 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:25.700 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:25.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:25.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:25.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:25.701 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:25.701 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:25.708 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:25.708 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:25.708 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:25.709 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:25.709 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:25.709 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:25.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:25.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:25.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:25.714 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:25.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:25.715 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:25.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:25.715 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:25.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:25.716 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:25Z" 2022-07-18T03:22:25.716 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:25.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:25Z" } 2022-07-18T03:22:25.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:25.719 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:25.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:25.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:30.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:30.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:30.722 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:30.773 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:30.773 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:30.849 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:30.850 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:30.850 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:30.851 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:30.851 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:30.851 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:30.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:30.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:30.852 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:30.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:30.853 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:30.853 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:30.853 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:30.854 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:30.854 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:30.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:30.854 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:30.855 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:30.855 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:30.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:30.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:30.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:30.856 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:30.856 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:30.856 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:30.857 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:30.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:30.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:30.858 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:30.858 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:30.859 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:30.859 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:30.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:30.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:30.860 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:30.860 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:30.861 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:30.861 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:30.861 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:30.861 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:30.862 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:30.862 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:30.863 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:30.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:30.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:30.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:30.864 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:30.864 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:30.864 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:30.864 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:30.865 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:30.865 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:30.865 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:30.865 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:30.866 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:30.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:30.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:30.866 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:30.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:30.867 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:30.867 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:30.867 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:30.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:30.868 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:30Z" 2022-07-18T03:22:30.868 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:30.870 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:30Z" } 2022-07-18T03:22:30.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:30.872 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:30.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:30.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:35.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:35.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:35.878 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:35.933 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:35.933 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:36.021 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:36.021 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:36.021 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:36.022 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:36.022 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:36.022 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:36.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:36.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:36.023 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:36.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:36.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:36.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:36.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:36.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:36.024 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:36.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:36.025 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:36.026 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:36.027 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:36.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:36.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:36.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:36.028 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:36.028 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:36.028 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:36.028 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:36.029 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:36.029 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:36.030 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:36.031 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:36.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:36.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:36.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:36.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:36.034 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:36.034 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:36.035 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:36.036 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:36.036 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:36.037 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:36.037 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:36.038 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:36.038 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:36.039 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:36.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:36.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:36.041 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:36.041 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:36.042 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:36.042 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:36.043 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:36.043 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:36.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:36.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:36.045 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:36.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:36.046 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:36.046 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:36.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:36.047 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:36.048 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:36.048 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:36.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:36.049 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:36Z" 2022-07-18T03:22:36.049 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:36.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:36Z" } 2022-07-18T03:22:36.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:36.052 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:36.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:36.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:41.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:41.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:41.057 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:41.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:41.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:41.191 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:41.191 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:41.191 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:41.192 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:41.192 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:41.192 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:41.192 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:41.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:41.193 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:41.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:41.193 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:41.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:41.194 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:41.194 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:41.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:41.195 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:41.195 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:41.195 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:41.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:41.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:41.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:41.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:41.197 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:41.197 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:41.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:41.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:41.198 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:41.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:41.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:41.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:41.198 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:41.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:41.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:41.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:41.199 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:41.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:41.200 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:41.200 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:41.200 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:41.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:41.201 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:41.201 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:41.201 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:41.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:41.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:41.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:41.202 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:41.203 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:41.203 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:41.203 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:41.203 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:41.204 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:41.204 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:41.204 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:41.204 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:41.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:41.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:41.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:41.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:41.206 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:41.206 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:41.206 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:41.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:41.207 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:41Z" 2022-07-18T03:22:41.207 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:41.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:41Z" } 2022-07-18T03:22:41.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:41.210 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:41.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:41.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:46.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:46.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:46.215 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:46.271 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:46.271 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:46.340 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:46.341 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:46.341 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:46.342 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:46.343 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:46.343 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:46.343 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:46.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:46.344 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:46.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:46.345 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:46.346 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:46.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:46.347 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:46.347 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:46.347 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:46.348 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:46.348 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:46.348 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:46.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:46.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:46.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:46.350 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:46.350 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:46.351 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:46.351 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:46.352 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:46.352 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:46.353 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:46.353 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:46.354 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:46.354 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:46.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:46.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:46.356 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:46.357 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:46.358 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:46.358 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:46.359 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:46.360 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:46.360 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:46.361 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:46.361 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:46.362 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:46.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:46.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:46.363 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:46.363 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:46.364 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:46.365 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:46.366 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:46.366 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:46.367 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:46.367 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:46.368 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:46.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:46.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:46.370 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:46.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:46.371 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:46.371 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:46.372 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:46.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:46.373 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:46Z" 2022-07-18T03:22:46.374 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:46.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:46Z" } 2022-07-18T03:22:46.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:46.379 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:46.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:46.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:51.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:51.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:51.387 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:51.462 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:51.463 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:51.555 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:51.556 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:51.556 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:51.556 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:51.556 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:51.557 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:51.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:51.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:51.557 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:51.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:51.558 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:51.558 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:51.559 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:51.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:51.559 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:51.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:51.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:51.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:51.561 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:51.561 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:51.562 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:51.563 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:51.563 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:51.563 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:51.564 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:51.564 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:51.564 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:51.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:51.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:51.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:51.565 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:51.565 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:51.566 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:51.566 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:51.566 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:51.567 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:51.567 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:51.567 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:51.567 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:51.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:51.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:51.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:51.568 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:51.569 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:51.569 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:51.569 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:51.571 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:51.571 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:51.572 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:51.572 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:51.573 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:51.573 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:51.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:51.576 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:51.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:51.577 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:51.578 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:51.578 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:51.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:51.579 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:51Z" 2022-07-18T03:22:51.579 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:51.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:51Z" } 2022-07-18T03:22:51.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:51.582 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:51.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:51.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:22:56.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:56.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:22:56.590 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:22:56.639 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:22:56.639 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:22:56.711 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:22:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:22:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:22:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:22:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:22:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:22:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:22:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:22:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:22:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:22:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:22:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:22:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:22:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:22:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:22:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:22:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:22:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:22:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:22:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:22:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:22:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:22:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:22:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:22:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:22:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:22:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:22:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:22:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:22:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:22:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:22:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:22:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:22:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:22:56Z" 2022-07-18T03:22:56.731 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:22:56.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:22:56Z" } 2022-07-18T03:22:56.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:22:56.733 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:22:56.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:22:56.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:01.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:01.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:01.737 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:01.792 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:01.793 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:01.877 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:01.878 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:01.878 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:01.878 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:01.879 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:01.879 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:01.879 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:01.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:01.880 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:01.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:01.880 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:01.881 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:01.881 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:01.881 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:01.882 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:01.882 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:01.882 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:01.883 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:01.883 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:01.883 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:01.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:01.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:01.884 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:01.884 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:01.885 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:01.885 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:01.885 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:01.885 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:01.886 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:01.886 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:01.886 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:01.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:01.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:01.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:01.887 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:01.887 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:01.888 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:01.888 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:01.888 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:01.889 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:01.889 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:01.889 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:01.889 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:01.889 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:01.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:01.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:01.890 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:01.890 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:01.891 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:01.891 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:01.891 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:01.891 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:01.892 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:01.892 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:01.892 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:01.892 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:01.893 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:01.893 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:01.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:01.893 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:01.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:01.894 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:01.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:01.894 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:01Z" 2022-07-18T03:23:01.895 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:01.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:01Z" } 2022-07-18T03:23:01.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:01.897 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:01.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:01.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:06.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:06.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:06.900 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:06.956 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:06.957 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:07.029 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:07.029 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:07.029 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:07.030 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:07.030 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:07.031 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:07.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:07.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:07.032 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:07.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:07.032 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:07.032 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:07.033 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:07.033 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:07.033 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:07.034 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:07.034 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:07.034 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:07.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:07.035 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:07.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:07.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:07.036 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:07.036 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:07.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:07.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:07.038 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:07.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:07.038 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:07.039 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:07.039 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:07.039 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:07.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:07.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:07.040 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:07.040 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:07.040 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:07.041 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:07.041 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:07.041 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:07.041 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:07.042 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:07.042 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:07.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:07.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:07.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:07.043 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:07.043 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:07.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:07.044 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:07.044 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:07.044 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:07.045 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:07.045 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:07.045 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:07.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:07.046 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:07.046 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:07.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:07.046 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:07.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:07.047 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:07.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:07.048 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:07Z" 2022-07-18T03:23:07.048 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:07.050 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:07Z" } 2022-07-18T03:23:07.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:07.052 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:07.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:07.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:12.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:12.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:12.056 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:12.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:12.114 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:12.183 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:12.184 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:12.185 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:12.185 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:12.186 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:12.186 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:12.187 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:12.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:12.188 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:12.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:12.189 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:12.189 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:12.190 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:12.190 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:12.191 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:12.191 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:12.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:12.192 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:12.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:12.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:12.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:12.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:12.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:12.195 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:12.196 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:12.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:12.197 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:12.197 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:12.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:12.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:12.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:12.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:12.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:12.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:12.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:12.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:12.201 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:12.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:12.201 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:12.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:12.202 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:12.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:12.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:12.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:12.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:12.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:12.204 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:12.204 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:12.204 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:12.204 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:12.205 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:12.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:12.205 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:12.205 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:12.206 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:12.206 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:12.206 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:12.206 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:12.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:12.207 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:12.207 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:12.207 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:12.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:12.208 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:12Z" 2022-07-18T03:23:12.208 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:12.209 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:12Z" } 2022-07-18T03:23:12.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:12.211 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:12.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:12.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:17.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:17.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:17.215 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:17.263 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:17.264 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:17.327 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:17.328 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:17.328 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:17.328 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:17.329 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:17.329 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:17.329 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:17.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:17.330 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:17.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:17.331 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:17.332 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:17.332 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:17.332 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:17.333 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:17.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:17.334 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:17.334 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:17.335 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:17.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:17.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:17.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:17.336 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:17.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:17.337 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:17.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:17.338 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:17.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:17.338 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:17.339 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:17.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:17.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:17.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:17.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:17.344 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:17.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:17.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:17.345 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:17.345 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:17.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:17.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:17.346 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:17.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:17.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:17.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:17.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:17.347 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:17.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:17.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:17.348 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:17.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:17.349 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:17.350 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:17.350 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:17.351 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:17.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:17.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:17.352 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:17.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:17.353 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:17.354 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:17.354 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:17.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:17.355 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:17Z" 2022-07-18T03:23:17.356 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:17.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:17Z" } 2022-07-18T03:23:17.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:17.361 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:17.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:17.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:22.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:22.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:22.367 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:22.423 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:22.424 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:22.509 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:22Z" 2022-07-18T03:23:22.530 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:22.532 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:22Z" } 2022-07-18T03:23:22.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:22.534 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:22.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:22.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:27.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:27.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:27.538 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:27.593 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:27.593 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:27.686 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:27.686 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:27.686 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:27.687 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:27.687 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:27.687 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:27.687 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:27.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:27.688 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:27.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:27.688 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:27.689 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:27.689 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:27.689 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:27.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:27.690 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:27.690 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:27.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:27.690 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:27.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:27.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:27.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:27.691 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:27.692 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:27.692 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:27.692 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:27.692 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:27.693 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:27.693 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:27.693 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:27.693 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:27Z" 2022-07-18T03:23:27.703 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:27.704 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:27Z" } 2022-07-18T03:23:27.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:27.705 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:27.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:27.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:32.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:32.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:32.710 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:32.772 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:32.773 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:32.926 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:32.926 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:32.927 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:32.927 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:32.927 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:32.928 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:32.928 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:32.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:32.928 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:32.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:32.929 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:32.929 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:32.929 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:32.930 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:32.930 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:32.930 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:32.930 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:32.931 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:32.931 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:32.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:32.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:32.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:32.932 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:32.932 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:32.932 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:32.933 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:32.933 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:32.933 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:32.933 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:32.934 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:32.934 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:32.934 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:32.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:32.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:32.935 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:32.935 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:32.936 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:32.936 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:32.936 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:32.936 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:32.937 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:32.937 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:32.937 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:32.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:32.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:32.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:32.938 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:32.938 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:32.939 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:32.939 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:32.939 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:32.939 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:32.940 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:32.940 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:32.940 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:32.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:32.941 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:32.941 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:32.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:32.941 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:32Z" 2022-07-18T03:23:32.943 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:32.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:32Z" } 2022-07-18T03:23:32.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:32.945 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:32.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:32.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:37.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:37.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:37.949 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:38.006 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:38.007 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:38.080 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:38.081 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:38.082 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:38.082 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:38.083 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:38.083 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:38.085 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:38.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:38.085 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:38.086 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:38.086 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:38.087 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:38.087 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:38.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:38.088 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:38.088 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:38.089 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:38.089 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:38.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:38.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:38.090 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:38.090 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:38.091 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:38.091 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:38.091 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:38.091 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:38.092 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:38.092 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:38.092 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:38.092 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:38.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:38.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:38.093 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:38.093 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:38Z" 2022-07-18T03:23:38.100 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:38.102 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:38Z" } 2022-07-18T03:23:38.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:38.103 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:38.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:38.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:43.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:43.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:43.107 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:43.162 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:43.163 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:43.235 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:43.235 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:43.235 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:43.236 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:43.236 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:43.237 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:43.237 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:43.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:43.240 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:43.240 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:43.240 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:43.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:43.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:43.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:43.241 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:43.241 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:43.242 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:43.242 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:43.242 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:43.242 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:43.243 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:43.243 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:43.243 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:43.243 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:43.251 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:43.251 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:43.251 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:43.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:43.252 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:43Z" 2022-07-18T03:23:43.252 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:43.253 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:43Z" } 2022-07-18T03:23:43.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:43.254 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:43.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:43.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:48.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:48.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:48.259 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:48.314 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:48.314 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:48.376 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:48.377 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:48.377 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:48.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:48.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:48.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:48.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:48.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:48.383 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:48.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:48.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:48.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:48.384 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:48.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:48.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:48.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:48.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:48.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:48.387 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:48.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:48.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:48.387 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:48.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:48.388 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:48.388 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:48.388 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:48.389 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:48.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:48.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:48.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:48.389 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:48.390 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:48.390 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:48.390 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:48.390 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:48.391 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:48.391 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:48.391 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:48.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:48.392 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:48.392 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:48.392 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:48.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:48.393 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:48.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:48.393 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:48.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:48.394 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:48Z" 2022-07-18T03:23:48.394 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:48.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:48Z" } 2022-07-18T03:23:48.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:48.396 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:48.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:48.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:53.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:53.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:53.400 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:53.456 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:53.456 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:53.536 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:53.537 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:53.537 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:53.538 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:53.538 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:53.539 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:53.539 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:53.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:53.539 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:53.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:53.540 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:53.540 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:53.541 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:53.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:53.541 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:53.541 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:53.542 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:53.542 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:53.542 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:53.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:53.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:53.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:53.543 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:53.544 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:53.544 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:53.544 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:53.545 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:53.545 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:53.546 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:53.546 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:53.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:53.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:53.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:53.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:53.549 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:53.549 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:53.550 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:53.550 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:53.551 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:53.551 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:53.552 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:53.552 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:53.552 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:53.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:53.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:53.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:53.553 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:53.554 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:53.554 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:53.554 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:53.554 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:53.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:53.555 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:53.555 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:53.555 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:53.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:53.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:53.556 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:53.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:53.557 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:53.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:53.557 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:53.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:53.558 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:53Z" 2022-07-18T03:23:53.558 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:53.560 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:53Z" } 2022-07-18T03:23:53.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:53.561 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:53.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:53.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:23:58.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:58.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:23:58.566 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:23:58.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:23:58.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:23:58.709 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:23:58.709 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:23:58.709 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:23:58.710 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:23:58.710 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:23:58.710 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:23:58.711 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:58.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:58.711 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:23:58.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:58.712 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:23:58.712 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:58.712 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:58.712 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:23:58.713 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:58.713 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:23:58.713 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:58.713 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:23:58.714 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:58.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:58.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:58.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:58.715 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:23:58.715 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:58.715 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:58.715 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:23:58.716 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:58.716 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:58.716 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:58.716 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:23:58.717 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:58.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:58.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:58.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:58.718 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:23:58.718 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:58.718 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:58.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:23:58.719 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:58.719 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:58.719 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:58.719 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:23:58.720 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:58.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:58.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:58.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:23:58.721 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:23:58.721 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:23:58.722 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:23:58.722 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:23:58.723 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:23:58.723 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:23:58.723 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:23:58.723 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:23:58.724 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:23:58.724 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:58.724 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:23:58.725 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:23:58.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:58.725 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:23:58.725 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:23:58.726 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:23:58.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:23:58.726 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:23:58Z" 2022-07-18T03:23:58.727 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:23:58.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:23:58Z" } 2022-07-18T03:23:58.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:23:58.729 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:23:58.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:23:58.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:03.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:03.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:03.733 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:03.787 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:03.787 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:03.947 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:03.948 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:03.948 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:03.949 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:03.949 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:03.950 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:03.950 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:03.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:03.951 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:03.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:03.952 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:03.953 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:03.953 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:03.954 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:03.954 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:03.954 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:03.955 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:03.955 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:03.956 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:03.956 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:03.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:03.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:03.957 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:03.958 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:03.959 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:03.960 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:03.960 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:03.960 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:03.960 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:03.961 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:03.961 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:03.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:03.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:03.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:03.962 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:03.962 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:03.963 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:03.963 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:03.963 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:03.963 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:03.963 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:03.964 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:03.964 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:03.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:03.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:03.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:03.965 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:03.965 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:03.965 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:03.966 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:03.966 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:03.966 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:03.966 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:03.967 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:03.967 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:03.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:03.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:03.968 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:03.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:03.968 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:03.968 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:03.969 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:03.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:03.969 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:03Z" 2022-07-18T03:24:03.970 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:03.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:03Z" } 2022-07-18T03:24:03.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:03.972 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:03.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:03.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:08.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:08.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:08.976 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:09.029 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:09.029 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:09.102 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:09.103 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:09.103 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:09.103 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:09.104 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:09.104 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:09.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:09.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:09.105 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:09.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:09.105 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:09Z" 2022-07-18T03:24:09.121 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:09.122 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:09Z" } 2022-07-18T03:24:09.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:09.123 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:09.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:09.124 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:14.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:14.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:14.128 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:14.178 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:14.179 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:14.256 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:14.257 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:14.257 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:14.257 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:14.257 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:14.258 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:14.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:14.258 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:14.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:14.259 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:14.259 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:14.259 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:14.259 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:14.260 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:14.260 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:14.260 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:14.260 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:14.261 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:14.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:14.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:14.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:14.261 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:14.262 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:14.262 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:14.262 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:14.263 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:14.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:14.263 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:14.263 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:14.263 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:14.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:14.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:14.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:14.264 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:14.265 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:14.265 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:14.265 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:14.265 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:14.266 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:14.266 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:14.266 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:14.266 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:14.267 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:14.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:14.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:14.267 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:14.268 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:14.268 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:14.268 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:14.268 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:14.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:14.269 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:14.269 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:14.269 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:14.270 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:14.270 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:14.270 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:14.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:14.271 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:14.271 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:14.271 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:14.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:14.271 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:14Z" 2022-07-18T03:24:14.272 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:14.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:14Z" } 2022-07-18T03:24:14.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:14.274 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:14.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:14.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:19.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:19.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:19.278 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:19.336 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:19.337 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:19.420 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:19.421 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:19.421 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:19.421 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:19.422 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:19.422 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:19.422 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:19.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:19.423 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:19.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:19.423 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:19.424 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:19.424 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:19.424 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:19.425 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:19.425 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:19.425 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:19.426 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:19.426 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:19.426 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:19.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:19.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:19.427 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:19.427 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:19.428 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:19.428 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:19.428 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:19.428 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:19.429 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:19.429 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:19.429 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:19.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:19.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:19.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:19.430 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:19.431 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:19.431 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:19.431 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:19.431 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:19.432 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:19.432 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:19.432 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:19.433 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:19.433 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:19.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:19.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:19.434 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:19.434 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:19.434 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:19.434 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:19.435 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:19.435 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:19.435 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:19.436 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:19.436 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:19.436 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:19.437 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:19.437 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:19.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:19.437 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:19.438 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:19.438 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:19.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:19.439 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:19Z" 2022-07-18T03:24:19.439 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:19.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:19Z" } 2022-07-18T03:24:19.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:19.442 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:19.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:19.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:24.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:24.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:24.446 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:24.503 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:24.503 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:24.566 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:24.566 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:24.567 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:24.567 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:24.567 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:24.567 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:24.568 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:24.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:24.568 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:24.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:24.569 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:24.569 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:24.569 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:24.569 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:24.570 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:24.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:24.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:24.570 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:24.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:24.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:24.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:24.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:24.572 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:24.572 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:24.572 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:24.572 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:24.572 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:24.573 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:24.573 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:24.573 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:24.573 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:24.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:24.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:24.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:24.574 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:24.575 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:24.575 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:24.575 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:24.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:24.576 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:24.576 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:24.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:24.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:24.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:24.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:24.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:24.577 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:24.578 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:24.578 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:24.578 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:24.579 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:24.579 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:24.579 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:24.579 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:24.580 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:24.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:24.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:24.581 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:24.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:24.581 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:24.581 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:24.582 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:24.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:24.583 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:24Z" 2022-07-18T03:24:24.583 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:24.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:24Z" } 2022-07-18T03:24:24.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:24.585 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:24.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:24.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:29.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:29.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:29.590 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:29.652 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:29.652 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:29.757 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:29.758 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:29.758 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:29.758 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:29.759 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:29.759 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:29.759 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:29.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:29.760 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:29.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:29.760 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:29.760 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:29.761 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:29.761 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:29.761 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:29Z" 2022-07-18T03:24:29.775 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:29.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:29Z" } 2022-07-18T03:24:29.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:29.778 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:29.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:29.779 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:34.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:34.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:34.784 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:34.836 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:34.836 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:34.916 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:34.917 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:34.917 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:34.918 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:34.918 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:34.918 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:34.918 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:34.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:34.919 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:34.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:34.919 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:34.920 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:34.920 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:34.920 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:34.920 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:34.921 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:34.921 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:34.921 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:34.921 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:34.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:34.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:34.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:34.922 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:34.923 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:34.923 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:34.923 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:34.923 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:34.924 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:34.924 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:34.924 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:34.924 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:34.925 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:34.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:34.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:34.925 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:34.926 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:34.926 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:34.926 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:34.927 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:34.927 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:34.927 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:34.927 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:34.928 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:34.928 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:34.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:34.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:34.929 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:34.929 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:34.929 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:34.929 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:34.930 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:34.930 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:34.930 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:34.930 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:34.931 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:34.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:34.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:34.931 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:34.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:34.932 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:34.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:34.932 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:34.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:34.933 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:34Z" 2022-07-18T03:24:34.933 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:34.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:34Z" } 2022-07-18T03:24:34.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:34.935 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:34.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:34.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:39.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:39.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:39.939 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:39.988 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:39.989 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:40.079 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:40.079 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:40.079 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:40.080 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:40.080 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:40.080 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:40.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:40.082 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:40.082 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:40.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:40.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:40.083 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:40.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:40.083 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:40.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:40.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:40.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:40.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:40.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:40.085 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:40.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:40.085 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:40.086 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:40.087 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:40.087 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:40.087 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:40.088 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:40.088 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:40.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:40.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:40.090 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:40.090 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:40.090 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:40.091 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:40.091 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:40.091 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:40.091 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:40.092 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:40.092 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:40.092 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:40.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:40.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:40.093 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:40.093 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:40.094 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:40.094 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:40.094 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:40.095 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:40.095 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:40.095 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:40.095 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:40.096 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:40.096 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:40.096 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:40.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:40.097 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:40.097 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:40.098 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:40.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:40.098 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:40Z" 2022-07-18T03:24:40.099 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:40.099 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:40Z" } 2022-07-18T03:24:40.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:40.101 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:40.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:40.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:45.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:45.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:45.105 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:45.161 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:45.162 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:45.225 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:45.225 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:45.225 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:45.226 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:45.226 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:45.226 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:45.227 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:45.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:45.227 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:45.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:45.228 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:45.228 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:45.228 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:45.229 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:45.229 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:45.229 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:45.229 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:45.230 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:45.230 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:45.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:45.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:45.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:45.231 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:45.231 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:45.231 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:45.232 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:45.232 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:45.232 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:45.232 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:45.233 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:45.233 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:45.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:45.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:45.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:45.234 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:45.234 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:45.234 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:45.235 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:45.235 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:45.235 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:45.235 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:45.236 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:45.236 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:45.236 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:45.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:45.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:45.237 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:45.237 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:45.237 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:45.238 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:45.238 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:45.238 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:45.238 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:45.239 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:45.239 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:45.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:45.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:45.240 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:45.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:45.240 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:45.240 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:45.241 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:45.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:45.241 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:45Z" 2022-07-18T03:24:45.241 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:45.243 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:45Z" } 2022-07-18T03:24:45.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:45.244 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:45.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:45.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:50.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:50.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:50.248 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:50.304 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:50.343 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:50.377 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:50.378 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:50.378 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:50.378 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:50.379 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:50.379 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:50.379 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:50.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:50.380 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:50.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:50.380 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:50.381 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:50.381 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:50.381 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:50.381 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:50.382 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:50.382 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:50.382 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:50.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:50.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:50.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:50.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:50.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:50.384 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:50.384 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:50.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:50.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:50.385 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:50.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:50.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:50.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:50.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:50.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:50.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:50.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:50.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:50.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:50.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:50.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:50.389 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:50.390 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:50.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:50.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:50.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:50.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:50.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:50.393 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:50.393 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:50.394 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:50.394 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:50.395 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:50.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:50.396 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:50.396 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:50.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:50.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:50.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:50.399 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:50.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:50.399 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:50.400 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:50.400 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:50.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:50.401 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:50Z" 2022-07-18T03:24:50.402 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:50.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:50Z" } 2022-07-18T03:24:50.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:50.406 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:50.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:50.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:24:55.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:55.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:24:55.411 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:24:55.466 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:24:55.467 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:24:55.553 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:24:55.554 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:24:55.554 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:24:55.554 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:24:55.554 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:24:55.555 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:24:55.555 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:55.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:55.555 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:24:55.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:55.556 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:24:55.556 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:55.556 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:55.557 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:24:55.557 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:55.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:24:55.558 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:55.558 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:24:55.558 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:55.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:55.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:55.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:55.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:24:55.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:55.560 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:55.560 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:24:55.560 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:55.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:55.561 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:55.561 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:24:55.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:55.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:55.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:55.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:55.562 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:24:55.563 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:55.563 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:55.564 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:24:55.564 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:55.565 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:55.565 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:55.565 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:24:55.565 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:55.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:55.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:55.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:24:55.566 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:24:55.567 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:24:55.567 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:24:55.567 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:24:55.567 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:24:55.568 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:24:55.568 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:24:55.568 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:24:55.568 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:24:55.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:55.569 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:24:55.569 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:24:55.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:55.570 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:24:55.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:24:55.570 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:24:55.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:24:55.571 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:24:55Z" 2022-07-18T03:24:55.571 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:24:55.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:24:55Z" } 2022-07-18T03:24:55.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:24:55.574 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:24:55.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:24:55.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:00.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:00.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:00.579 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:00.632 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:00.633 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:00.712 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:00.713 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:00.713 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:00.714 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:00.714 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:00.714 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:00.715 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:00.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:00.716 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:00.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:00.716 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:00.716 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:00.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:00.717 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:00.717 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:00.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:00.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:00.718 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:00.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:00.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:00.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:00.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:00.720 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:00.720 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:00.721 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:00.721 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:00.721 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:00.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:00.722 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:00.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:00.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:00.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:00.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:00.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:00.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:00.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:00.724 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:00.724 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:00.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:00.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:00.725 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:00.725 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:00.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:00.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:00.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:00.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:00.727 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:00.728 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:00.728 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:00.729 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:00.729 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:00.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:00.730 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:00.731 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:00.731 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:00.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:00.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:00.732 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:00.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:00.733 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:00.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:00.733 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:00.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:00.734 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:00Z" 2022-07-18T03:25:00.734 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:00.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:00Z" } 2022-07-18T03:25:00.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:00.737 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:00.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:00.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:05.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:05.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:05.741 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:05.797 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:05.798 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:05.883 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:05.883 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:05.884 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:05.884 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:05.884 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:05.884 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:05.885 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:05.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:05.885 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:05.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:05.886 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:05.886 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:05.886 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:05.886 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:05.887 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:05.887 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:05.887 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:05.887 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:05.888 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:05.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:05.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:05.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:05.889 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:05.889 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:05.889 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:05.889 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:05.890 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:05.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:05.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:05.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:05.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:05.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:05.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:05.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:05.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:05.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:05.892 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:05.892 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:05.893 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:05.893 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:05.893 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:05.893 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:05.894 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:05.894 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:05.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:05.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:05.895 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:05.895 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:05.895 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:05.895 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:05.896 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:05.896 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:05.896 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:05.896 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:05.897 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:05.897 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:05.897 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:05.897 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:05.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:05.898 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:05.898 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:05.898 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:05.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:05.899 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:05Z" 2022-07-18T03:25:05.899 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:05.900 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:05Z" } 2022-07-18T03:25:05.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:05.901 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:05.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:05.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:10.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:10.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:10.906 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:10.964 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:10.965 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:11.025 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:11.026 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:11.026 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:11.027 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:11.027 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:11.027 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:11.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:11.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:11.028 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:11.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:11.029 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:11.029 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:11.030 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:11.030 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:11.031 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:11.031 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:11.031 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:11.032 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:11.032 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:11.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:11.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:11.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:11.033 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:11.033 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:11.033 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:11.034 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:11.034 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:11.035 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:11.035 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:11.035 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:11.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:11.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:11.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:11.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:11.037 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:11.037 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:11.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:11.038 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:11.038 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:11.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:11.038 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:11.039 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:11.039 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:11.039 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:11.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:11.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:11.040 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:11.040 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:11.041 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:11.041 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:11.041 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:11.041 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:11.042 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:11.042 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:11.042 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:11.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:11.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:11.043 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:11.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:11.044 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:11.044 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:11.045 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:11.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:11.046 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:11Z" 2022-07-18T03:25:11.046 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:11.049 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:11Z" } 2022-07-18T03:25:11.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:11.050 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:11.051 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:11.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:16.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:16.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:16.056 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:16.112 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:16.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:16.197 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:16.198 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:16.198 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:16.199 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:16.199 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:16.199 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:16.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:16.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:16.200 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:16.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:16.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:16.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:16.201 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:16.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:16.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:16.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:16.202 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:16.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:16.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:16.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:16.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:16.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:16.204 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:16.204 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:16.204 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:16.204 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:16.205 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:16.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:16.205 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:16.205 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:16.206 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:16.206 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:16.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:16.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:16.207 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:16.207 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:16.207 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:16.207 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:16.208 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:16.209 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:16.209 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:16.210 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:16.211 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:16.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:16.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:16.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:16.212 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:16.212 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:16.212 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:16.213 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:16.214 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:16.214 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:16.215 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:16.215 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:16.216 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:16.216 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:16.217 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:16.217 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:16.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:16.218 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:16.219 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:16.220 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:16.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:16.220 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:16Z" 2022-07-18T03:25:16.220 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:16.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:16Z" } 2022-07-18T03:25:16.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:16.223 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:16.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:16.224 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:21.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:21.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:21.228 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:21.276 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:21.276 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:21.331 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:21.331 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:21.332 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:21.332 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:21.332 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:21.332 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:21.333 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:21.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:21.333 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:21.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:21.334 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:21.334 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:21.334 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:21.335 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:21.335 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:21.335 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:21.335 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:21.336 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:21.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:21.336 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:21.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:21.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:21.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:21.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:21.337 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:21.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:21.338 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:21.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:21.339 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:21.339 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:21.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:21.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:21.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:21.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:21.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:21.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:21.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:21.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:21.341 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:21.341 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:21.341 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:21.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:21.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:21.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:21.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:21.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:21.343 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:21.343 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:21.343 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:21.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:21.344 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:21.344 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:21.344 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:21.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:21.345 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:21.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:21.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:21.346 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:21.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:21.346 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:21.346 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:21.347 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:21.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:21.347 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:21Z" 2022-07-18T03:25:21.347 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:21.348 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:21Z" } 2022-07-18T03:25:21.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:21.350 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:21.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:21.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:26.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:26.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:26.354 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:26.420 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:26.421 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:26.565 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:26.565 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:26.566 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:26.566 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:26.567 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:26.567 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:26.567 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:26.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:26.568 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:26.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:26.568 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:26.568 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:26.569 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:26.569 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:26.569 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:26.569 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:26.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:26.570 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:26.570 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:26.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:26.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:26.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:26.571 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:26.571 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:26.572 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:26.572 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:26.572 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:26.572 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:26.572 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:26.573 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:26.573 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:26.573 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:26.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:26.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:26.574 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:26.574 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:26.575 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:26.575 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:26.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:26.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:26.576 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:26.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:26.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:26.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:26.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:26.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:26.577 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:26.577 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:26.578 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:26.578 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:26.578 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:26.578 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:26.579 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:26.579 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:26.579 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:26.579 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:26.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:26.580 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:26.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:26.580 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:26.581 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:26.581 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:26.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:26.581 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:26Z" 2022-07-18T03:25:26.581 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:26.583 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:26Z" } 2022-07-18T03:25:26.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:26.584 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:26.584 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:26.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:31.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:31.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:31.588 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:31.645 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:31.645 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:31.710 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:31.710 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:31.711 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:31.711 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:31.711 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:31.712 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:31.712 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:31.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:31.713 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:31.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:31.713 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:31.714 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:31.714 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:31.715 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:31.716 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:31.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:31.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:31.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:31.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:31.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:31.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:31.718 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:31.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:31.718 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:31.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:31.719 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:31.719 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:31.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:31.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:31.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:31.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:31.721 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:31.721 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:31.721 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:31.721 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:31.722 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:31.722 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:31.722 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:31.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:31.723 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:31.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:31.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:31.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:31.724 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:31.724 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:31.724 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:31.725 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:31.725 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:31.725 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:31.725 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:31.726 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:31.726 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:31.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:31.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:31.727 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:31.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:31.727 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:31.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:31.728 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:31.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:31.729 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:31Z" 2022-07-18T03:25:31.729 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:31.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:31Z" } 2022-07-18T03:25:31.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:31.732 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:31.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:31.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:36.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:36.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:36.735 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:36.791 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:36.791 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:36.869 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:36.869 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:36.870 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:36.870 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:36.870 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:36.871 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:36.871 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:36.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:36.872 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:36.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:36.872 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:36.872 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:36.873 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:36.873 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:36.873 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:36.874 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:36.874 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:36.874 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:36.874 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:36.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:36.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:36.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:36.876 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:36.876 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:36.876 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:36.876 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:36.877 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:36.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:36.877 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:36.877 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:36.878 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:36.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:36.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:36.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:36.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:36.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:36.879 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:36.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:36.880 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:36.880 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:36.880 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:36.881 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:36.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:36.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:36.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:36.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:36.882 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:36.882 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:36.883 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:36.883 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:36.884 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:36.885 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:36.885 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:36.886 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:36.886 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:36.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:36.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:36.888 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:36.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:36.889 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:36.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:36.890 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:36.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:36.891 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:36Z" 2022-07-18T03:25:36.892 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:36.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:36Z" } 2022-07-18T03:25:36.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:36.895 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:36.896 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:36.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:41.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:41.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:41.899 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:41.957 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:41.958 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:42.046 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:42Z" 2022-07-18T03:25:42.066 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:42.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:42Z" } 2022-07-18T03:25:42.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:42.069 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:42.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:42.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:47.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:47.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:47.074 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:47.123 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:47.124 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:47.178 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:47.179 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:47.179 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:47.179 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:47.179 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:47.180 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:47.180 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:47.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:47.180 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:47.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:47.181 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:47.181 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:47.182 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:47.182 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:47.182 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:47.183 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:47.183 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:47.183 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:47.183 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:47.184 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:47.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:47.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:47.184 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:47.185 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:47.185 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:47.185 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:47.185 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:47.186 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:47.186 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:47.186 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:47.186 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:47.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:47.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:47.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:47.187 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:47.188 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:47.188 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:47.188 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:47.189 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:47.189 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:47.189 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:47.189 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:47.190 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:47.190 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:47.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:47.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:47.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:47.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:47.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:47.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:47.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:47.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:47.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:47.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:47.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:47.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:47.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:47.194 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:47.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:47.194 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:47.194 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:47.195 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:47.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:47.195 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:47Z" 2022-07-18T03:25:47.195 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:47.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:47Z" } 2022-07-18T03:25:47.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:47.198 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:47.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:47.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:52.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:52.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:52.201 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:52.257 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:52.257 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:52.332 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:52.333 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:52.333 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:52.333 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:52.333 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:52.334 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:52.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:52.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:52.334 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:52.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:52.335 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:52.335 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:52.335 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:52.336 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:52.336 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:52.336 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:52.336 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:52.337 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:52.337 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:52.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:52.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:52.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:52.338 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:52.338 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:52.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:52.339 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:52.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:52.339 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:52.339 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:52.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:52.340 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:52.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:52.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:52.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:52.341 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:52.341 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:52.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:52.342 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:52.342 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:52.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:52.342 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:52.343 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:52.343 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:52.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:52.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:52.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:52.344 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:52.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:52.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:52.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:52.345 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:52.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:52.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:52.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:52.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:52.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:52.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:52.346 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:52.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:52.347 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:52.347 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:52.348 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:52.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:52.348 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:52Z" 2022-07-18T03:25:52.348 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:52.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:52Z" } 2022-07-18T03:25:52.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:52.351 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:52.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:52.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:25:57.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:57.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:25:57.355 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:25:57.411 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:25:57.411 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:25:57.491 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:25:57.492 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:25:57.492 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:25:57.493 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:25:57.493 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:25:57.493 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:25:57.493 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:57.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:57.494 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:25:57.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:57.494 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:25:57.495 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:57.495 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:57.495 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:25:57.495 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:57.496 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:25:57.496 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:57.496 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:25:57.497 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:57.497 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:57.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:57.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:57.498 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:25:57.498 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:57.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:57.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:25:57.499 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:57.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:57.499 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:57.500 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:25:57.500 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:57.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:57.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:57.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:57.501 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:25:57.501 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:57.501 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:57.502 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:25:57.502 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:57.503 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:57.503 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:57.503 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:25:57.504 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:57.504 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:57.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:57.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:25:57.505 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:25:57.505 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:25:57.505 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:25:57.505 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:25:57.506 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:25:57.506 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:25:57.506 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:25:57.506 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:25:57.507 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:25:57.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:57.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:25:57.508 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:25:57.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:57.508 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:25:57.509 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:25:57.509 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:25:57.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:25:57.509 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:25:57Z" 2022-07-18T03:25:57.510 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:25:57.511 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:25:57Z" } 2022-07-18T03:25:57.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:25:57.512 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:25:57.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:25:57.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:02.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:02.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:02.517 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:02.573 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:02.573 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:02.655 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:02.655 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:02.655 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:02.656 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:02.656 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:02.656 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:02.657 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:02.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:02.657 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:02.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:02.658 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:02.658 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:02.658 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:02.658 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:02.659 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:02.659 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:02.659 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:02.659 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:02.660 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:02.660 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:02.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:02.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:02.661 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:02.661 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:02.661 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:02.662 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:02.662 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:02.662 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:02.662 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:02.663 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:02.663 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:02.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:02.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:02.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:02.664 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:02.665 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:02.665 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:02.665 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:02.665 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:02.666 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:02.666 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:02.666 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:02.666 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:02.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:02.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:02.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:02.667 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:02.668 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:02.668 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:02.668 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:02.668 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:02.669 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:02.669 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:02.669 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:02.670 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:02.670 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:02.670 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:02.671 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:02.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:02.672 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:02.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:02.673 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:02.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:02.674 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:02Z" 2022-07-18T03:26:02.674 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:02.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:02Z" } 2022-07-18T03:26:02.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:02.679 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:02.679 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:02.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:07.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:07.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:07.684 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:07.739 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:07.740 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:07.808 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:07.808 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:07.809 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:07.809 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:07.809 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:07.809 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:07.810 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:07.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:07.810 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:07.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:07.811 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:07.811 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:07.812 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:07.812 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:07.813 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:07.813 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:07.813 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:07.814 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:07.814 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:07.814 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:07.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:07.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:07.815 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:07.815 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:07.816 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:07.816 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:07.816 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:07.817 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:07.817 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:07.817 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:07.817 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:07.818 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:07.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:07.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:07.818 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:07.819 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:07.819 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:07.819 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:07.819 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:07.820 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:07.820 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:07.820 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:07.821 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:07.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:07.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:07.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:07.822 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:07.822 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:07.823 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:07.824 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:07.824 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:07.825 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:07.825 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:07.826 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:07.826 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:07.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:07.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:07.828 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:07.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:07.829 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:07.829 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:07.830 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:07.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:07.830 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:07Z" 2022-07-18T03:26:07.831 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:07.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:07Z" } 2022-07-18T03:26:07.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:07.834 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:07.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:07.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:12.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:12.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:12.839 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:12.894 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:12.895 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:12.968 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:12.968 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:12.969 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:12.969 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:12.969 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:12.970 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:12.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:12.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:12.971 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:12.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:12.971 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:12.973 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:12.973 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:12.973 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:12.973 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:12.974 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:12.974 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:12.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:12.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:12.975 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:12.975 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:12.976 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:12.976 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:12.977 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:12.977 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:12.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:12.978 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:12.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:12.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:12.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:12.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:12.979 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:12.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:12.979 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:12.980 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:12.980 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:12.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:12.980 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:12.981 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:12.981 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:12.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:12.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:12.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:12.982 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:12.982 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:12.983 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:12.983 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:12.983 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:12.984 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:12.984 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:12.985 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:12.986 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:12.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:12.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:12.987 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:12.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:12.988 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:12.989 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:12.989 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:12.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:12.991 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:12Z" 2022-07-18T03:26:12.991 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:12.993 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:12Z" } 2022-07-18T03:26:12.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:12.995 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:12.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:12.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:17.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:17.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:18.000 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:18.056 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:18.056 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:18.132 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:18.132 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:18.133 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:18.133 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:18.133 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:18.134 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:18.134 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:18.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:18.135 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:18.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:18.136 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:18.136 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:18.136 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:18.137 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:18.137 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:18.137 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:18.137 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:18.138 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:18.138 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:18.138 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:18.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:18.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:18.139 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:18.139 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:18.141 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:18.141 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:18.142 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:18.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:18.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:18.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:18.143 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:18.143 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:18.144 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:18.144 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:18.144 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:18.145 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:18.145 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:18.145 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:18.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:18.146 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:18.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:18.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:18.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:18.147 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:18.147 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:18.147 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:18.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:18.148 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:18.148 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:18.148 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:18.149 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:18.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:18.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:18.149 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:18.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:18.150 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:18.150 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:18.150 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:18.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:18.151 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:18Z" 2022-07-18T03:26:18.151 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:18.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:18Z" } 2022-07-18T03:26:18.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:18.154 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:18.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:18.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:23.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:23.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:23.158 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:23.214 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:23.214 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:23.303 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:23.309 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:23.309 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:23.313 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:23.313 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:23.313 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:23.313 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:23.314 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:23.314 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:23.314 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:23.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:23.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:23.315 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:23.315 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:23.315 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:23.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:23.319 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:23Z" 2022-07-18T03:26:23.319 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:23.320 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:23Z" } 2022-07-18T03:26:23.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:23.322 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:23.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:23.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:28.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:28.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:28.325 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:28.373 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:28.374 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:28.457 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:28.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:28.474 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:28.474 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:28.474 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:28Z" 2022-07-18T03:26:28.475 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:28.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:28Z" } 2022-07-18T03:26:28.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:28.478 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:28.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:28.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:33.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:33.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:33.482 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:33.529 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:33.530 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:33.595 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:33Z" 2022-07-18T03:26:33.613 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:33.614 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:33Z" } 2022-07-18T03:26:33.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:33.615 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:33.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:33.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:38.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:38.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:38.619 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:38.675 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:38.675 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:38.747 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:38.748 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:38.748 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:38.748 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:38.749 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:38.749 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:38.749 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:38.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:38.750 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:38.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:38.750 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:38.751 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:38.751 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:38.751 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:38.751 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:38.752 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:38.752 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:38.752 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:38.752 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:38.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:38.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:38.763 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:38.763 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:38.766 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:38Z" 2022-07-18T03:26:38.766 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:38.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:38Z" } 2022-07-18T03:26:38.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:38.768 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:38.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:38.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:43.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:43.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:43.772 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:43.828 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:43.829 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:43.912 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:43.912 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:43.913 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:43.913 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:43.913 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:43.913 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:43.914 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:43.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:43.915 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:43.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:43.916 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:43.916 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:43.916 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:43.917 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:43.917 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:43.933 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:43Z" 2022-07-18T03:26:43.933 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:43.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:43Z" } 2022-07-18T03:26:43.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:43.935 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:43.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:43.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:48.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:48.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:48.939 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:48.997 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:48.997 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:49.062 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:49.063 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:49.063 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:49.063 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:49.063 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:49.064 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:49.064 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:49.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:49.064 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:49.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:49.065 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:49.065 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:49.065 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:49.066 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:49.066 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:49.066 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:49.067 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:49.067 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:49.067 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:49.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:49.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:49.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:49.068 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:49.068 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:49.068 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:49.069 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:49.069 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:49.069 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:49.069 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:49.070 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:49.070 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:49.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:49Z" 2022-07-18T03:26:49.078 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:49.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:49Z" } 2022-07-18T03:26:49.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:49.081 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:49.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:49.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:54.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:54.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:54.085 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:54.133 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:54.133 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:54.203 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:54.203 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:54.204 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:54.204 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:54.204 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:54.204 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:54.205 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:54.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:54.205 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:54.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:54.206 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:54.206 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:54.206 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:54.207 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:54.207 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:54.207 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:54.208 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:54.208 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:54.208 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:54.208 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:54.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:54.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:54.209 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:54.209 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:54.210 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:54.210 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:54.210 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:54.211 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:54.211 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:54.211 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:54.211 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:54.212 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:54.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:54.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:54.212 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:54.213 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:54.213 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:54.213 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:54.213 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:54.214 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:54.214 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:54.214 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:54.215 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:54.215 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:54.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:54.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:54.216 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:54.216 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:54.216 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:54.216 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:54.217 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:54.217 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:54.217 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:54.217 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:54.218 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:54.218 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:54.218 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:54.218 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:54.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:54.219 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:54.219 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:54.219 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:54.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:54.220 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:54Z" 2022-07-18T03:26:54.221 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:54.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:54Z" } 2022-07-18T03:26:54.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:54.223 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:54.224 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:54.224 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:26:59.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:59.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:26:59.227 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:26:59.280 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:26:59.281 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:26:59.380 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:26:59.380 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:26:59.381 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:26:59.381 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:26:59.382 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:26:59.382 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:26:59.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:26:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:26:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:26:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:26:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:26:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:26:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:26:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:26:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:59.391 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:26:59.391 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:59.391 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:26:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:26:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:26:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:26:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:26:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:26:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:26:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:26:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:26:59.397 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:26:59.397 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:26:59.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:26:59.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:59.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:26:59.398 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:26:59.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:59.399 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:26:59.399 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:26:59.399 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:26:59.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:26:59.400 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:26:59Z" 2022-07-18T03:26:59.400 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:26:59.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:26:59Z" } 2022-07-18T03:26:59.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:26:59.403 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:26:59.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:26:59.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:04.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:04.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:04.407 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:04.462 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:04.463 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:04.534 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:04.535 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:04.535 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:04.535 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:04.536 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:04.536 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:04.536 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:04.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:04.537 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:04.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:04.538 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:04.538 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:04.538 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:04.539 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:04.539 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:04.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:04.540 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:04.540 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:04.540 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:04.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:04.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:04.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:04.541 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:04.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:04.542 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:04.542 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:04.542 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:04.543 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:04.543 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:04.543 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:04.543 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:04.545 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:04.545 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:04.545 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:27:04.546 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:04.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:04.546 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:04.546 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:27:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:04.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:04.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:04.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:04.548 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:04.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:04.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:04.549 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:04.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:04.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:04.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:04.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:04.551 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:04.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:04.551 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:04.551 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:04.552 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:04.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:04.552 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:04Z" 2022-07-18T03:27:04.553 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:04.554 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:04Z" } 2022-07-18T03:27:04.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:04.555 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:04.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:04.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:09.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:09.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:09.560 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:09.616 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:09.616 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:09.677 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:09.677 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:09.678 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:09.678 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:09.678 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:09.678 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:09.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:09.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:09.679 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:09.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:09.680 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:09.680 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:09.680 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:09.681 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:09.681 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:09.681 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:09.681 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:09.682 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:09.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:09.682 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:09.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:09.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:09.683 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:09.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:09.683 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:09.684 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:09.684 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:09.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:09.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:09.685 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:09.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:09.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:09.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:09.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:09.686 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:09.686 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:09.686 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:09.687 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:27:09.687 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:09.687 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:09.687 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:09.688 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:27:09.688 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:09.688 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:09.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:09.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:09.689 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:09.689 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:09.689 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:09.690 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:09.690 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:09.690 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:09.690 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:09.691 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:09.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:09.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:09.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:09.692 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:09.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:09.692 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:09.692 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:09.692 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:09.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:09.693 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:09Z" 2022-07-18T03:27:09.693 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:09.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:09Z" } 2022-07-18T03:27:09.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:09.696 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:09.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:09.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:14.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:14.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:14.700 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:14.755 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:14.756 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:14.847 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:14.847 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:14.848 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:14.848 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:14.848 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:14.849 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:14.849 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:14.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:14.849 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:14.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:14.850 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:14.850 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:14.850 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:14.851 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:14.851 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:14.851 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:14.851 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:14.852 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:14.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:14.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:14.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:14.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:14.853 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:14.853 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:14.853 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:14.854 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:14.854 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:14.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:14.854 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:14.855 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:14.855 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:14.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:14.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:14.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:14.856 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:14.856 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:14.857 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:14.857 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:27:14.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:14.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:14.858 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:14.858 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:27:14.858 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:14.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:14.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:14.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:14.859 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:14.859 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:14.860 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:14.860 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:14.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:14.860 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:14.861 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:14.861 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:14.861 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:14.861 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:14.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:14.862 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:14.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:14.863 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:14.863 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:14.863 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:14.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:14.864 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:14Z" 2022-07-18T03:27:14.864 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:14.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:14Z" } 2022-07-18T03:27:14.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:14.866 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:14.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:14.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:19.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:19.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:19.870 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:19.925 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:19.926 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:20.007 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:20.008 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:20.008 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:20.008 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:20.009 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:20.009 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:20.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:20.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:20.010 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:20.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:20.010 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:20.011 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:20.011 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:20.011 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:20.011 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:20.012 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:20.012 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:20.012 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:20.012 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:20.013 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:20.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:20.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:20.013 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:20.014 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:20.014 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:20.014 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:20.015 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:20.015 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:20.015 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:20.015 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:20.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:20.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:20.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:20.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:20.016 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:20.017 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:20.017 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:20.017 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:27:20.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:20.018 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:20.018 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:20.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:27:20.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:20.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:20.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:20.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:20.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:20.020 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:20.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:20.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:20.021 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:20.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:20.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:20.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:20.023 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:20.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:20.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:20.023 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:20.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:20.024 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:20.024 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:20.024 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:20.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:20.025 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:19Z" 2022-07-18T03:27:20.025 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:20.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:19Z" } 2022-07-18T03:27:20.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:20.028 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:20.029 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:20.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:25.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:25.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:25.032 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:25.083 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:25.083 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:25.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:25.144 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:25.145 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:25.145 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:25.145 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:25.146 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:25.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:25.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:25.146 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:25.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:25.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:25.147 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:25.147 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:25.148 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:25.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:25.148 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:25.149 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:25.149 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:25.149 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:25.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:25.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:25.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:25.150 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:25.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:25.151 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:25.151 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:25.152 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:25.152 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:25.152 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:25.152 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:25.153 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:25.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:25.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:25.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:25.154 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:25.154 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:25.154 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:25.154 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114540.200245_130.1", 2022-07-18T03:27:25.155 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:25.155 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:25.155 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:25.155 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:22:20.200245Z", 2022-07-18T03:27:25.156 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:25.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:25.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:25.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:25.157 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:25.157 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:25.157 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:25.157 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:25.158 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:25.158 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:25.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:25.159 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:25.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:25.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:25.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:25.160 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:25.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:25.160 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:25.160 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:25.161 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:25.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:25.161 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:25Z" 2022-07-18T03:27:25.161 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:25.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114540.200245_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:22:20.200245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:25Z" } 2022-07-18T03:27:25.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114540.200245_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:22:20.200245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:25.164 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:25.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114540.200245_130.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:25.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:30.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... FAIL 2022-07-18T03:27:30.174 DEBUG:boto:path=/mlvank-19/ 2022-07-18T03:27:30.175 DEBUG:boto:auth_path=/mlvank-19/ 2022-07-18T03:27:30.176 DEBUG:boto:Method: PUT 2022-07-18T03:27:30.176 DEBUG:boto:Path: /mlvank-19/ 2022-07-18T03:27:30.177 DEBUG:boto:Data: 2022-07-18T03:27:30.177 DEBUG:boto:Headers: {} 2022-07-18T03:27:30.178 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:27:30.178 DEBUG:boto:Port: 8000 2022-07-18T03:27:30.179 DEBUG:boto:Params: {} 2022-07-18T03:27:30.180 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:27:30.181 DEBUG:boto:Token: None 2022-07-18T03:27:30.181 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:27:30 GMT /mlvank-19/ 2022-07-18T03:27:30.182 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:cMZ0QbM2bmw3fD7RK6ihpp/ztCY= 2022-07-18T03:27:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:27:30 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:cMZ0QbM2bmw3fD7RK6ihpp/ztCY=', 'Content-Length': '0'} 2022-07-18T03:27:30.202 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7f44b256f155757-0062d4d322-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:27:30 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:27:30.203 DEBUG:rgw_multi.tests:created bucket=mlvank-19 2022-07-18T03:27:30.204 DEBUG:boto:path=/mlvank-19/ 2022-07-18T03:27:30.204 DEBUG:boto:auth_path=/mlvank-19/ 2022-07-18T03:27:30.205 DEBUG:boto:path=/mlvank-19/?versioning 2022-07-18T03:27:30.205 DEBUG:boto:auth_path=/mlvank-19/?versioning 2022-07-18T03:27:30.205 DEBUG:boto:Method: PUT 2022-07-18T03:27:30.206 DEBUG:boto:Path: /mlvank-19/?versioning 2022-07-18T03:27:30.206 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:27:30.207 DEBUG:boto:Headers: {} 2022-07-18T03:27:30.207 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:27:30.208 DEBUG:boto:Port: 8000 2022-07-18T03:27:30.208 DEBUG:boto:Params: {} 2022-07-18T03:27:30.209 DEBUG:boto:Token: None 2022-07-18T03:27:30.209 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:27:30 GMT /mlvank-19/?versioning 2022-07-18T03:27:30.210 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:vyUE6GOFFAztiKWO8QkbRqqEFvw= 2022-07-18T03:27:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:27:30 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:vyUE6GOFFAztiKWO8QkbRqqEFvw=', 'Content-Length': '228'} 2022-07-18T03:27:30.217 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e3942e749cb9722-0062d4d322-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:27:30 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:27:30.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:30.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:30.219 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:30.272 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:30.272 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:30.318 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:27:30.341 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:27:30.342 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:27:30.342 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:30.342 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:27:30.343 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:27:30.343 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:27:30.343 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:27:30.343 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:27:30.344 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:27:30.344 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:27:30.344 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:30.344 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:27:30.215685Z" 2022-07-18T03:27:30.345 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:27:30.345 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:27:30.345 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:27:30.345 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:27:30.346 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:27:30.346 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:27:30.347 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:30.348 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:27:30.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:30.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:30.349 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:30.405 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:30.406 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:30.465 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:27:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:30Z" 2022-07-18T03:27:30.482 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:30.483 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:30Z" } 2022-07-18T03:27:30.484 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:30.484 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:30.485 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:30.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:35.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:35.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:35.489 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:35.544 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:35.545 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:35.612 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:35.613 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:35.613 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:35.614 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:35.615 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:35.615 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:35.616 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:35.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:35.617 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:35.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:35.618 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:35.618 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:35.619 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:35.619 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:35.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:35.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:35.621 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:35.621 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:35.622 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:35.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:35.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:35.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:35.624 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:35.624 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:35.625 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:35.625 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:35.625 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:35.626 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:35.627 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:35.627 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:35.628 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:35.628 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:35.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:35.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:35.630 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:35.630 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:35.631 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:35.631 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:35.632 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:35.632 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:35.632 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:35.633 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:27:35.633 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:35.633 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:35.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:35.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:35.634 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:35.634 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:35.634 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:35.635 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:35.635 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:35.636 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:35.636 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:35.636 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:35.637 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:35.637 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:35.637 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:35.638 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:35.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:35.638 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:35.639 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:35.639 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:35.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:35.639 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:35Z" 2022-07-18T03:27:35.640 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:35.641 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:35Z" } 2022-07-18T03:27:35.642 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:35.643 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:35.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:35.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:40.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:40.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:40.650 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:40.718 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:40.719 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:40.795 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:40.796 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:40.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:40.798 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:40.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:40.798 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:40.799 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:40.799 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:40.799 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:40.799 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:40.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:40.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:40.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:40.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:40.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:40.802 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:40.803 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:40.803 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:40.803 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:40.804 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:40.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:40.804 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:40.804 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:40.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:40.806 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:40.806 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:40.806 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:40.807 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:40.807 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:40.807 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:40.808 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:40.808 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:27:40.808 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:40.808 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:40.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:40.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:40.809 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:40.809 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:40.810 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:40.810 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:40.810 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:40.811 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:40.811 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:40.811 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:40.811 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:40.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:40.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:40.812 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:40.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:40.813 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:40.813 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:40.813 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:40.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:40.813 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:40Z" 2022-07-18T03:27:40.814 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:40.815 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:40Z" } 2022-07-18T03:27:40.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:40.816 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:40.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:40.817 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:45.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:45.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:45.820 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:45.875 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:45.875 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:45.939 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:45.939 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:45.940 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:45.940 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:45.940 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:45.940 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:45.941 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:45.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:45.941 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:45.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:45.942 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:45.942 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:45.943 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:45.943 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:45.943 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:45.943 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:45.944 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:45.944 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:45.944 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:45.944 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:45.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:45.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:45.945 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:45.945 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:45.946 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:45.946 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:45.946 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:27:45.951 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:45.951 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:45Z" 2022-07-18T03:27:45.957 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:45.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:45Z" } 2022-07-18T03:27:45.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:45.960 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:45.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:45.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:50.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:50.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:50.963 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:51.031 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:51.031 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:51.180 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:51.180 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:51.181 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:51.181 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:51.181 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:51.182 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:51.182 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:51.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:51.183 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:51.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:51.183 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:27:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:51Z" 2022-07-18T03:27:51.205 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:51.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:51Z" } 2022-07-18T03:27:51.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:51.207 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:51.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:51.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:27:56.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:56.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:27:56.211 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:27:56.266 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:27:56.267 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:27:56.334 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:27:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:27:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:27:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:27:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:27:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:27:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:27:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:27:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:27:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:27:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:27:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:27:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:27:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:27:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:27:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:27:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:27:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:27:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:27:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:27:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:27:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:27:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:27:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:27:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:27:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:27:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:27:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:27:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:27:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:27:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:27:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:27:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:27:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:27:56Z" 2022-07-18T03:27:56.351 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:27:56.352 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:27:56Z" } 2022-07-18T03:27:56.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:27:56.354 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:27:56.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:27:56.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:01.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:01.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:01.358 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:01.406 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:01.406 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:01.471 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:01.471 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:01.472 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:01.472 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:01.472 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:01.473 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:01.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:01.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:01.473 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:01.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:01.474 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:01.474 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:01.474 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:01.475 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:01.475 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:01.475 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:01.475 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:01.476 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:01.476 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:01.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:01.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:01.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:01.477 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:01.477 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:01.477 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:01.478 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:01.478 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:01.478 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:01.478 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:01.479 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:01.479 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:01.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:01.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:01.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:01.480 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:01.480 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:01.480 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:01.481 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:01.481 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:01.481 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:01.481 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:01.482 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:01.482 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:01.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:01.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:01.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:01.483 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:01.483 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:01.484 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:01.484 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:01.484 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:01.484 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:01.485 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:01.485 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:01.485 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:01.485 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:01.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:01.486 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:01.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:01.486 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:01.487 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:01.487 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:01.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:01.487 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:01Z" 2022-07-18T03:28:01.488 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:01.489 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:01Z" } 2022-07-18T03:28:01.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:01.490 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:01.491 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:01.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:06.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:06.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:06.494 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:06.544 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:06.546 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:06.599 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:06.600 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:06.600 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:06.601 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:06.601 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:06.602 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:06.603 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:06.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:06.604 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:06.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:06.604 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:06.605 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:06.605 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:06.605 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:06.605 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:06.606 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:06.606 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:06.606 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:06.607 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:06.607 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:06.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:06.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:06.608 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:06.608 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:06.608 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:06.609 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:06.609 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:06.609 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:06.610 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:06.610 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:06.610 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:06.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:06.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:06.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:06.611 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:06.612 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:06.612 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:06.612 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:06.613 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:06.613 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:06.613 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:06.613 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:06.614 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:06.614 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:06.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:06.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:06.615 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:06.616 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:06.618 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:06.618 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:06.619 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:06.619 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:06.620 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:06.620 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:06.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:06.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:06.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:06.622 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:06.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:06.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:06.624 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:06Z" 2022-07-18T03:28:06.625 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:06.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:06Z" } 2022-07-18T03:28:06.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:06.628 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:06.629 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:06.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:11.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:11.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:11.632 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:11.688 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:11.688 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:11.759 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:11.760 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:11.760 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:11.760 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:11.761 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:11.761 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:11.761 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:11.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:11.762 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:11.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:11.762 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:11.763 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:11.763 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:11.763 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:11.763 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:11.764 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:11.764 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:11.764 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:11.765 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:11.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:11.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:11.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:11.766 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:11.766 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:11.766 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:11.766 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:11.767 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:11.767 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:11.767 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:11.767 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:11.768 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:11.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:11.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:11.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:11.769 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:11.769 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:11.770 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:11.770 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:11.770 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:11.770 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:11.771 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:11.771 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:11.771 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:11.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:11.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:11.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:11.772 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:11.772 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:11.773 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:11.773 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:11.773 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:11.773 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:11.774 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:11.774 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:11.774 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:11.775 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:11.775 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:11.775 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:11.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:11.776 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:11.776 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:11.776 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:11.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:11.777 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:11Z" 2022-07-18T03:28:11.777 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:11.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:11Z" } 2022-07-18T03:28:11.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:11.780 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:11.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:11.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:16.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:16.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:16.785 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:16.851 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:16.852 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:16.964 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:16.964 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:16.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:16.967 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:16.967 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:16.967 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:16.967 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:16.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:16.968 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:16.968 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:16.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:16.971 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:16.971 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:16.971 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:16.972 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:16.972 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:16.972 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:16.972 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:16.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:16.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:16.973 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:16.973 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:16.974 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:16.974 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:16.974 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:16.974 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:16.975 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:16.975 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:16.975 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:16.975 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:16.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:16.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:16.976 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:16.976 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:16.977 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:16.977 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:16.977 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:16.977 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:16.978 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:16.978 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:16.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:16.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:16.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:16.979 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:16.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:16.980 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:16.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:16.980 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:16.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:16.981 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:16Z" 2022-07-18T03:28:16.981 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:16.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:16Z" } 2022-07-18T03:28:16.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:16.984 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:16.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:16.985 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:21.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:21.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:21.988 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:22.043 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:22.044 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:22.106 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:22.106 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:22.108 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:22.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:22.108 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:22.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:22.109 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:22.109 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:22.109 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:22.111 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:22.111 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:22.111 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:22.113 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:22.113 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:22.113 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:22.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:22.122 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:22.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:22.122 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:22.123 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:22.123 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:22.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:22.123 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:22Z" 2022-07-18T03:28:22.124 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:22.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:22Z" } 2022-07-18T03:28:22.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:22.126 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:22.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:22.127 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:27.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:27.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:27.130 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:27.182 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:27.183 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:27.249 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:27.250 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:27.250 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:27.250 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:27.251 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:27.251 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:27.251 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:27.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:27.259 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:27.259 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:27.259 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:27.266 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:27.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:27.266 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:27Z" 2022-07-18T03:28:27.267 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:27.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:27Z" } 2022-07-18T03:28:27.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:27.269 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:27.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:27.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:32.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:32.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:32.273 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:32.323 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:32.323 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:32.406 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:32.406 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:32.410 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:32.410 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:32.410 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:32.411 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:32.411 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:32.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:32.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:32.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:32.412 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:32.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:32.412 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:32.413 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:32.413 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:32.413 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:32.413 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:32.414 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:32.414 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:32.414 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:32.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:32.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:32.416 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:32.416 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:32.416 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:32.416 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:32.417 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:32.417 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:32.417 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:32.417 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:32.417 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:32.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:32.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:32.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:32.418 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:32.419 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:32.419 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:32.419 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:32.419 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:32.420 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:32.420 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:32.420 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:32.421 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:32.421 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:32.423 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:32.424 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:32.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:32.424 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:32.424 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:32.425 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:32.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:32.425 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:32Z" 2022-07-18T03:28:32.426 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:32.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:32Z" } 2022-07-18T03:28:32.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:32.428 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:32.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:32.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:37.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:37.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:37.432 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:37.481 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:37.481 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:37.557 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:37.557 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:37.558 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:37.558 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:37.559 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:37.559 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:37.559 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:37.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:37.560 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:37.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:37.561 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:37.561 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:37.561 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:37.562 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:37.562 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:37.562 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:37.562 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:37.563 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:37.563 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:37.563 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:37.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:37.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:37.564 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:37.564 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:37.565 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:37.565 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:37.566 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:37.566 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:37.567 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:37.567 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:37.568 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:37.569 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:37.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:37.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:37.570 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:37.571 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:37.571 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:37.572 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:37.572 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:37.572 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:37.572 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:37.573 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:37.573 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:37.573 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:37.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:37.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:37.574 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:37.574 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:37.575 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:37.575 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:37.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:37.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:37.576 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:37.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:37.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:37.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:37.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:37.577 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:37.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:37.577 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:37.578 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:37.578 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:37.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:37.578 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:37Z" 2022-07-18T03:28:37.579 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:37.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:37Z" } 2022-07-18T03:28:37.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:37.582 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:37.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:37.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:42.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:42.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:42.587 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:42.653 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:42.654 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:42.738 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:42Z" 2022-07-18T03:28:42.768 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:42.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:42Z" } 2022-07-18T03:28:42.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:42.773 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:42.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:42.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:47.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:47.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:47.778 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:47.827 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:47.827 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:47.897 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:47.898 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:47.898 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:47.898 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:47.898 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:47.899 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:47.899 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:47.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:47.899 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:47.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:47.900 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:47.900 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:47.901 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:47.901 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:47.901 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:47.901 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:47.902 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:47.902 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:47.903 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:47.903 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:47.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:47.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:47.904 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:47.904 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:47.904 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:47.904 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:47.904 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:47.905 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:47.905 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:47.905 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:47.906 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:47.906 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:47.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:47.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:47.907 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:47.907 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:47.907 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:47.907 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:47.908 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:47.908 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:47.908 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:47.908 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:47.909 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:47.909 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:47.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:47.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:47.910 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:47.910 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:47.911 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:47.911 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:47.911 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:47.911 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:47.912 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:47.912 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:47.912 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:47.912 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:47.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:47.913 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:47.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:47.913 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:47.914 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:47.914 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:47.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:47.915 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:47Z" 2022-07-18T03:28:47.915 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:47.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:47Z" } 2022-07-18T03:28:47.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:47.917 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:47.918 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:47.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:52.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:52.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:52.922 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:52.980 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:52.980 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:53.058 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:53.059 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:53.060 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:53.060 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:53.060 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:53.061 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:53.062 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:53.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:53.063 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:53.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:53.064 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:53.064 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:53.064 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:53.065 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:53.065 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:53.066 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:53.066 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:53.067 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:53.067 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:53.068 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:53.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:53.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:53.069 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:53.070 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:53.070 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:53.071 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:53.071 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:53.072 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:53.074 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:53.075 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:53.075 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:53.075 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:53.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:53.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:53.076 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:53.076 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:53.076 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:53.077 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:53.077 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:53.077 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:53.077 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:53.078 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:53.078 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:53.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:53.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:53.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:53.079 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:53.079 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:53.080 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:53.080 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:53.080 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:53.080 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:53.080 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:53.081 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:53.081 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:53.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:53.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:53.082 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:53.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:53.082 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:53.082 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:53.083 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:53.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:53.083 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:53Z" 2022-07-18T03:28:53.083 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:53.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:53Z" } 2022-07-18T03:28:53.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:53.086 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:53.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:53.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:28:58.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:58.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:28:58.090 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:28:58.139 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:28:58.139 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:28:58.203 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:28:58.204 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:28:58.204 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:28:58.204 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:28:58.204 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:28:58.205 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:28:58.205 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:58.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:58.205 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:28:58.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:58.206 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:28:58.206 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:58.206 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:58.207 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:28:58.207 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:58.207 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:28:58.207 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:58.208 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:28:58.208 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:58.208 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:58.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:58.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:58.209 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:28:58.209 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:58.210 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:58.210 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:28:58.210 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:58.210 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:58.211 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:58.211 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:28:58.211 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:58.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:58.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:58.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:58.212 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:28:58.212 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:58.213 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:58.213 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:28:58.213 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:58.213 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:58.214 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:58.214 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:28:58.214 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:58.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:58.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:58.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:28:58.215 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:28:58.215 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:28:58.216 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:28:58.216 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:28:58.216 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:28:58.216 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:28:58.217 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:28:58.217 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:28:58.217 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:28:58.217 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:58.218 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:28:58.218 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:28:58.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:58.218 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:28:58.219 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:28:58.219 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:28:58.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:28:58.219 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:28:58Z" 2022-07-18T03:28:58.220 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:28:58.221 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:28:58Z" } 2022-07-18T03:28:58.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:28:58.223 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:28:58.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:28:58.224 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:03.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:03.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:03.227 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:03.282 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:03.283 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:03.355 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:03.356 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:03.356 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:03.356 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:03.357 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:03.357 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:03.357 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:03.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:03.358 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:03.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:03.358 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:03.358 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:03.359 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:03.359 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:03.359 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:03.359 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:03.360 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:03.360 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:03.360 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:03.360 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:03.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:03.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:03.361 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:03.361 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:03.362 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:03.362 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:03.362 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:03.362 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:03.363 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:03.363 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:03.363 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:03.363 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:03.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:03.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:03.364 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:03.364 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:03.365 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:03.365 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:03.365 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:03.365 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:03.366 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:03.366 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:03.366 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:03.366 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:03.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:03.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:03.367 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:03.367 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:03.368 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:03.368 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:03.368 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:03.368 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:03.369 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:03.369 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:03.369 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:03.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:03.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:03.370 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:03.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:03.370 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:03.371 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:03.371 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:03.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:03.371 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:03Z" 2022-07-18T03:29:03.372 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:03.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:03Z" } 2022-07-18T03:29:03.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:03.374 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:03.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:03.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:08.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:08.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:08.378 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:08.433 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:08.434 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:08.505 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:08.506 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:08.506 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:08.507 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:08.507 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:08.507 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:08.507 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:08.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:08.508 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:08.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:08.508 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:08.509 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:08.509 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:08.509 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:08.509 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:08.510 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:08.511 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:08.511 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:08.511 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:08.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:08.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:08.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:08.512 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:08.512 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:08.513 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:08.513 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:08.513 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:08.513 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:08.514 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:08.514 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:08.514 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:08.515 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:08.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:08.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:08.515 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:08.515 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:08.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:08.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:08.516 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:08.516 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:08.517 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:08.517 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:08.517 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:08.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:08.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:08.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:08.518 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:08.519 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:08.519 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:08.519 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:08.519 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:08.520 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:08.520 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:08.520 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:08.520 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:08.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:08.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:08.521 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:08.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:08.522 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:08.522 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:08.522 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:08.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:08.523 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:08Z" 2022-07-18T03:29:08.523 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:08.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:08Z" } 2022-07-18T03:29:08.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:08.526 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:08.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:08.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:13.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:13.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:13.529 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:13.579 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:13.580 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:13.654 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:13.655 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:13.655 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:13.656 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:13.656 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:13.656 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:13.657 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:13.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:13.657 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:13.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:13.658 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:13.658 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:13.659 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:13.659 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:13.660 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:13.660 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:13.660 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:13.661 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:13.661 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:13.662 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:13.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:13.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:13.663 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:13.664 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:13.664 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:13.664 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:13.664 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:13.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:13.665 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:13.666 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:13.667 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:13.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:13.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:13.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:13.668 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:13.668 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:13.669 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:13.669 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:13.670 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:13.670 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:13.671 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:13.671 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:13.672 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:13.672 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:13.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:13.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:13.674 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:13.674 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:13.675 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:13.675 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:13.676 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:13.677 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:13.677 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:13.678 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:13.678 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:13.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:13.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:13.680 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:13.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:13.681 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:13.681 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:13.682 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:13.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:13.683 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:13Z" 2022-07-18T03:29:13.684 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:13.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:13Z" } 2022-07-18T03:29:13.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:13.688 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:13.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:13.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:18.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:18.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:18.693 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:18.749 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:18.750 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:18.841 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:18.842 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:18.843 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:18.844 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:18.845 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:18.846 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:18.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:18.848 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:18.849 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:18.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:18.851 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:18.852 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:18.853 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:18.853 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:18.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:18.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:18.854 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:18.854 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:18.854 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:18.855 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:18.855 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:18.855 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:18.855 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:18.856 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:18.856 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:18.856 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:18.856 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:18.857 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:18.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:18.857 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:18.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:18.858 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:18.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:18.858 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:18Z" 2022-07-18T03:29:18.858 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:18.860 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:18Z" } 2022-07-18T03:29:18.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:18.861 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:18.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:18.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:23.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:23.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:23.865 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:23.921 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:23.922 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:23.986 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:23.986 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:23.987 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:23.987 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:23.987 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:23.987 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:23.988 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:23.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:23.988 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:23.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:23.989 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:23.989 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:23.989 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:23.989 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:23.990 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:23.990 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:23.990 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:23.991 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:23.991 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:23.991 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:23.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:23.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:23.992 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:23.992 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:23.992 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:23.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:23.995 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:23.996 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:23.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:23.998 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:23.999 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:24.000 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:24.001 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:24.002 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:24.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:24.002 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:23Z" 2022-07-18T03:29:24.002 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:24.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:23Z" } 2022-07-18T03:29:24.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:24.005 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:24.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:24.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:29.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:29.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:29.010 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:29.066 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:29.066 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:29.153 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:29.153 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:29.154 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:29.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:29.156 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:29.157 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:29.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:29.159 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:29.160 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:29.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:29.162 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:29.163 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:29.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:29.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:29.165 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:29.165 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:29.165 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:29.166 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:29.166 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:29.166 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:29.166 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:29.167 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:29.167 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:29.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:29.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:29.168 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:29.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:29.168 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:29.168 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:29.169 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:29.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:29.169 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:29Z" 2022-07-18T03:29:29.169 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:29.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:29Z" } 2022-07-18T03:29:29.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:29.172 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:29.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:29.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:34.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:34.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:34.176 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:34.232 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:34.233 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:34.299 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:34.300 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:34.300 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:34.300 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:34.300 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:34.301 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:34.301 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:34.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:34.302 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:34.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:34.302 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:34.303 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:34.303 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:34.303 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:34.303 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:34.304 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:34.304 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:34.304 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:34.305 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:34.305 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:34.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:34.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:34.306 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:34.306 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:34.306 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:34.307 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:34.307 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:34.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:34.307 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:34.308 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:34.308 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:34.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:34.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:34.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:34.309 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:34.309 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:34.309 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:34.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:34.310 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:34.310 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:34.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:34.311 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:34.311 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:34.311 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:34.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:34.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:34.312 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:34.313 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:34.313 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:34.313 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:34.313 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:34.314 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:34.314 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:34.314 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:34.314 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:34.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:34.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:34.315 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:34.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:34.316 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:34.316 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:34.316 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:34.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:34.317 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:34Z" 2022-07-18T03:29:34.317 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:34.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:34Z" } 2022-07-18T03:29:34.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:34.320 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:34.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:34.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:39.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:39.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:39.324 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:39.380 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:39.381 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:39.464 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:39.464 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:39.464 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:39.464 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:39.465 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:39.465 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:39.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:39.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:39.466 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:39.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:39.466 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:39.467 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:39.467 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:39.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:39.467 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:39.468 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:39.468 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:39.468 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:39.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:39.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:39.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:39.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:39.469 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:39.469 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:39.470 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:39.470 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:39.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:39.471 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:39.471 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:39.471 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:39.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:39.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:39.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:39.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:39.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:39.473 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:39.473 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:39.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:39.473 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:39.473 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:39.474 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:39.474 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:39.474 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:39.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:39.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:39.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:39.475 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:39.476 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:39.476 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:39.476 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:39.476 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:39.476 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:39.477 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:39.477 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:39.477 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:39.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:39.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:39.478 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:39.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:39.478 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:39.479 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:39.479 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:39.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:39.479 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:39Z" 2022-07-18T03:29:39.480 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:39.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:39Z" } 2022-07-18T03:29:39.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:39.482 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:39.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:39.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:44.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:44.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:44.486 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:44.551 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:44.551 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:44.719 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:44.720 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:44.720 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:44.720 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:44.721 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:44.721 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:44.721 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:44.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:44.722 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:44.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:44.722 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:44.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:44.723 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:44.723 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:44.723 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:44.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:44.724 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:44.724 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:44.724 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:44.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:44.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:44.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:44.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:44.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:44.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:44.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:44.726 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:44.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:44.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:44.727 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:44.727 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:44.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:44.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:44.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:44.728 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:44.729 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:44.729 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:44.729 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:44.729 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:44.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:44.730 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:44.730 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:44.730 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:44.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:44.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:44.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:44.731 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:44.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:44.732 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:44.732 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:44.732 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:44.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:44.733 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:44.733 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:44.733 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:44.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:44.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:44.734 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:44.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:44.735 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:44.735 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:44.735 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:44.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:44.736 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:44Z" 2022-07-18T03:29:44.736 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:44.736 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:44Z" } 2022-07-18T03:29:44.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:44.738 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:44.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:44.739 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:49.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:49.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:49.742 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:49.797 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:49.798 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:49.861 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:49.862 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:49.862 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:49.862 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:49.863 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:49.863 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:49.863 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:49.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:49.864 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:49.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:49.864 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:49.865 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:49.865 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:49.866 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:49.866 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:49.866 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:49.866 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:49.867 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:49.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:49.867 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:49.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:49.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:49.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:49.868 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:49.869 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:49.869 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:49.869 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:49.869 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:49.870 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:49.870 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:49.870 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:49.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:49.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:49.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:49.871 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:49.872 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:49.872 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:49.872 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:49.872 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:49.873 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:49.873 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:49.873 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:49.873 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:49.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:49.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:49.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:49.874 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:49.875 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:49.875 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:49.875 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:49.875 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:49.876 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:49.876 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:49.876 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:49.877 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:49.877 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:49.877 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:49.877 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:49.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:49.878 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:49.878 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:49.878 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:49.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:49.879 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:49Z" 2022-07-18T03:29:49.879 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:49.881 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:49Z" } 2022-07-18T03:29:49.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:49.882 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:49.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:49.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:29:54.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:54.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:29:54.886 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:29:54.940 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:29:54.940 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:29:55.015 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:29:55.015 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:29:55.015 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:29:55.016 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:29:55.016 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:29:55.016 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:29:55.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:55.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:55.017 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:29:55.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:55.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:29:55.018 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:55.018 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:55.018 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:29:55.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:55.019 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:29:55.019 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:55.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:29:55.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:55.020 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:55.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:55.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:55.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:29:55.021 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:55.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:55.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:29:55.021 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:55.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:55.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:55.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:29:55.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:55.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:55.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:55.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:55.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:29:55.023 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:55.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:55.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:29:55.024 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:55.024 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:55.025 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:55.025 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:29:55.025 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:55.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:55.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:55.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:29:55.026 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:29:55.026 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:29:55.027 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:29:55.027 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:29:55.027 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:29:55.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:29:55.028 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:29:55.028 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:29:55.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:29:55.028 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:55.028 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:29:55.029 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:29:55.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:55.029 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:29:55.030 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:29:55.030 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:29:55.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:29:55.030 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:29:54Z" 2022-07-18T03:29:55.031 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:29:55.032 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:29:54Z" } 2022-07-18T03:29:55.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:29:55.033 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:29:55.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:29:55.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:00.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:00.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:00.037 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:00.093 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:00.093 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:00.183 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:00.183 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:00.184 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:00.184 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:00.184 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:00.185 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:00.185 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:00.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:00.186 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:00.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:00.187 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:00.187 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:00.188 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:00.188 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:00.189 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:00.189 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:00.190 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:00.190 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:00.191 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:00.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:00.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:00.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:00.192 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:00.192 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:00.192 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:00.193 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:00.193 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:00.193 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:00.193 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:00.194 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:00.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:00.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:00.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:00.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:00.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:00.195 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:00.195 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:00.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:00.196 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:00.196 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:00.196 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:00.197 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:00.197 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:00.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:00.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:00.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:00.198 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:00.198 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:00.198 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:00.199 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:00.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:00.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:00.199 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:00.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:00.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:00.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:00.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:00.201 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:00.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:00.201 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:00.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:00.202 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:00.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:00.202 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:00Z" 2022-07-18T03:30:00.202 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:00.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:00Z" } 2022-07-18T03:30:00.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:00.205 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:00.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:00.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:05.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:05.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:05.208 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:05.256 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:05.257 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:05.324 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:05.325 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:05.326 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:05.326 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:05.327 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:05.327 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:05.327 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:05.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:05.328 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:05.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:05.329 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:05.329 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:05.330 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:05.330 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:05.331 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:05.331 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:05.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:05.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:05.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:05.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:05.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:05.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:05.333 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:05.334 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:05.334 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:05.334 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:05.334 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:05.335 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:05.335 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:05.335 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:05.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:05.336 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:05.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:05.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:05.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:05.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:05.337 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:05.337 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:05.338 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:05.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:05.338 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:05.338 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:05.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:05.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:05.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:05.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:05.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:05.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:05.340 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:05.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:05.341 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:05.341 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:05.341 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:05.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:05.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:05.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:05.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:05.342 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:05.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:05.343 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:05.343 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:05.343 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:05.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:05.344 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:05Z" 2022-07-18T03:30:05.344 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:05.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:05Z" } 2022-07-18T03:30:05.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:05.347 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:05.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:05.347 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:10.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:10.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:10.350 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:10.402 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:10.402 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:10.471 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:10.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:10.480 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:10.481 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:10.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:10.483 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:10.484 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:10.484 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:10.484 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:10.484 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:10.485 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:10.485 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:10.485 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:10.485 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:10.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:10.486 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:10.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:10.486 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:10.487 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:10.487 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:10.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:10.487 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:10Z" 2022-07-18T03:30:10.488 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:10.489 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:10Z" } 2022-07-18T03:30:10.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:10.490 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:10.491 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:10.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:15.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:15.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:15.494 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:15.545 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:15.546 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:15.612 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:15.613 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:15.613 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:15.614 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:15.614 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:15.615 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:15.615 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:15.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:15.616 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:15.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:15.618 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:15.618 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:15.619 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:15.619 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:15.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:15.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:15.621 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:15.621 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:15.622 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:15.623 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:15.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:15.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:15.624 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:15.625 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:15.625 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:15.626 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:15.627 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:15.627 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:15.628 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:15.628 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:15.629 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:15.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:15.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:15.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:15.631 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:15.632 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:15.633 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:15Z" 2022-07-18T03:30:15.644 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:15.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:15Z" } 2022-07-18T03:30:15.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:15.647 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:15.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:15.648 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:20.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:20.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:20.651 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:20.707 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:20.708 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:20.770 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:20.770 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:20.770 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:20.771 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:20.771 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:20.772 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:20.772 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:20.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:20.773 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:20.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:20.773 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:20.774 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:20.774 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:20.774 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:20.774 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:20.775 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:20.775 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:20.775 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:20.775 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:20.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:20.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:20.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:20.776 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:20.777 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:20.777 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:20.777 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:20.777 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:20.778 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:20.778 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:20.778 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:20.779 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:20.779 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:20.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:20.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:20.780 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:20.780 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:20.781 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:20.781 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:20.782 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:20.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:20.782 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:20.783 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:20.783 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:20.783 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:20.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:20.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:20.784 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:20.784 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:20.784 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:20.785 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:20.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:20.785 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:20.785 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:20.786 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:20.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:20.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:20.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:20.787 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:20.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:20.788 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:20.788 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:20.788 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:20.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:20.789 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:20Z" 2022-07-18T03:30:20.789 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:20.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:20Z" } 2022-07-18T03:30:20.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:20.793 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:20.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:20.794 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:25.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:25.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:25.797 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:25.853 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:25.853 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:25.940 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:25.941 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:25.942 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:25.942 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:25.943 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:25.943 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:25.944 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:25.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:25.945 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:25.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:25.946 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:25.946 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:25.946 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:25.947 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:25.947 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:25.948 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:25.948 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:25.952 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:25.952 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:25.952 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:25.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:25Z" 2022-07-18T03:30:25.962 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:25.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:25Z" } 2022-07-18T03:30:25.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:25.966 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:25.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:25.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:30.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:30.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:30.970 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:31.020 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:31.021 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:31.083 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:31.084 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:31.084 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:31.085 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:31.085 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:31.085 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:31.085 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:31.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:31.086 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:31.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:31.086 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:31.087 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:31.087 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:31.087 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:31.088 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:31.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:31.088 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:31.088 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:31.089 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:31.089 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:31.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:31.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:31.090 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:31.090 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:31.090 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:31.091 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:31.091 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:31.091 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:31.091 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:31.092 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:31.092 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:31.092 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:31.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:31.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:31.093 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:31.093 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:31.093 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:31.094 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:31.094 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:31.094 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:31.094 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:31.095 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:31.095 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:31.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:31.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:31.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:31.096 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:31.096 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:31.096 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:31.097 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:31.097 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:31.097 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:31.097 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:31.097 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:31.098 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:31.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:31.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:31.098 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:31.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:31.099 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:31.099 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:31.099 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:31.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:31.100 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:31Z" 2022-07-18T03:30:31.100 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:31.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:31Z" } 2022-07-18T03:30:31.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:31.103 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:31.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:31.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:36.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:36.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:36.108 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:36.165 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:36.166 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:36.233 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:36.234 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:36.234 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:36.234 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:36.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:36.236 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:36.237 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:36.238 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:36.238 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:36.239 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:36.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:36.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:36.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:36.240 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:36.241 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:36.241 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:36.241 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:36.241 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:36.242 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:36.242 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:36.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:36.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:36.243 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:36.243 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:36.243 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:36.243 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:36.244 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:36.244 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:36.244 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:36.245 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:36.245 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:36.246 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:36.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:36.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:36.247 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:36.247 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:36.248 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:36.248 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:36.249 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:36.249 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:36.250 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:36.250 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:36.251 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:36.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:36.252 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:36.252 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:36.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:36.253 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:36.253 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:36.254 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:36.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:36.255 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:36Z" 2022-07-18T03:30:36.255 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:36.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:36Z" } 2022-07-18T03:30:36.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:36.259 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:36.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:36.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:41.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:41.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:41.262 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:41.318 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:41.318 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:41.381 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:41.381 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:41.381 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:41.382 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:41.382 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:41.382 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:41.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:41.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:41.383 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:41.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:41.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:41.384 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:41.384 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:41.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:41.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:41.385 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:41.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:41.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:41.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:41.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:41.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:41.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:41.387 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:41.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:41.388 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:41.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:41.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:41.389 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:41.389 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:41.389 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:41.390 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:41.390 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:41.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:41.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:41.391 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:41.391 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:41.391 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:41.392 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:41.392 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:41.392 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:41.392 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:41.393 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:41.393 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:41.393 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:41.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:41.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:41.394 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:41.394 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:41.394 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:41.395 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:41.395 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:41.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:41.395 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:41.396 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:41.396 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:41.396 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:41.396 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:41.397 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:41.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:41.397 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:41.397 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:41.398 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:41.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:41.398 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:41Z" 2022-07-18T03:30:41.398 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:41.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:41Z" } 2022-07-18T03:30:41.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:41.401 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:41.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:41.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:46.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:46.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:46.405 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:46.460 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:46.461 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:46.555 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:46.555 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:46.556 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:46.556 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:46.556 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:46.557 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:46.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:46.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:46.558 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:46.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:46.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:46.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:46.559 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:46.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:46.560 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:46.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:46.560 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:46.561 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:46.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:46.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:46.563 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:46.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:46.565 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:46.566 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:46.566 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:46.567 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:46.568 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:46.568 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:46.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:46.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:46.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:46.569 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:46.569 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:46.569 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:46.570 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:46.570 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:46.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:46.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:46.571 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:46.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:46.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:46.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:46.572 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:46.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:46.572 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:46.573 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:46.573 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:46.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:46.573 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:46Z" 2022-07-18T03:30:46.574 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:46.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:46Z" } 2022-07-18T03:30:46.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:46.576 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:46.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:46.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:51.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:51.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:51.581 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:51.637 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:51.637 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:51.711 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:51.711 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:51.722 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:51.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:51.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:51.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:51.724 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:51.725 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:51.725 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:51.725 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:51.725 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:51.726 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:51.726 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:51.726 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:51.726 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:51.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:51.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:51.727 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:51.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:51.728 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:51.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:51.728 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:51.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:51.729 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:51Z" 2022-07-18T03:30:51.729 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:51.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:51Z" } 2022-07-18T03:30:51.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:51.731 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:51.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:51.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:30:56.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:56.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:30:56.735 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:30:56.784 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:30:56.785 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:30:56.839 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:30:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:30:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:30:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:30:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:30:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:30:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:30:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:56.842 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:30:56.842 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:56.842 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:56.842 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:30:56.843 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:56.843 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:30:56.843 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:56.843 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:30:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:30:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:30:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:30:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:30:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:30:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:30:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:30:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:30:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:30:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:30:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:30:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:30:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:30:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:30:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:30:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:30:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:30:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:30:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:30:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:30:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:30:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:30:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:30:56Z" 2022-07-18T03:30:56.856 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:30:56.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:30:56Z" } 2022-07-18T03:30:56.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:30:56.858 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:30:56.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:30:56.859 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:01.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:01.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:01.862 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:01.916 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:01.917 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:01.993 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:01.994 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:01.994 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:01.994 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:01.995 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:01.995 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:01.995 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:01.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:01.996 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:01.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:01.996 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:01.997 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:01.997 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:01.997 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:01.997 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:01.998 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:01.998 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:01.998 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:01.998 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:01.999 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:01.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:01.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:01.999 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:02.000 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:02.000 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:02.000 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:02.000 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:02.001 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:02.001 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:02.001 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:02.002 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:02.002 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:02.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:02.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:02.003 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:02.003 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:02.003 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:02.003 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:02.004 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:02.004 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:02.004 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:02.004 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:02.005 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:02.005 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:02.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:02.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:02.006 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:02.006 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:02.006 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:02.007 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:02.007 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:02.007 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:02.007 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:02.008 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:02.008 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:02.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:02.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:02.009 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:02.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:02.009 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:02.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:02.010 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:02.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:02.010 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:01Z" 2022-07-18T03:31:02.011 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:02.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:01Z" } 2022-07-18T03:31:02.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:02.013 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:02.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:02.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:07.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:07.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:07.016 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:07.072 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:07.072 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:07.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:07.145 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:07.145 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:07.145 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:07.145 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:07.146 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:07.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:07.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:07.147 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:07.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:07.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:07.148 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:07.148 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:07.148 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:07.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:07.149 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:07.149 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:07.150 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:07.150 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:07.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:07.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:07.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:07.151 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:07.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:07.152 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:07.152 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:07.152 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:07.153 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:07.153 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:07.153 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:07.154 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:07.154 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:07.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:07.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:07.155 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:07.155 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:07.155 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:07.155 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:07.156 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:07.156 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:07.156 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:07.156 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:07.157 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:07.157 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:07.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:07.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:07.158 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:07.158 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:07.158 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:07.159 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:07.159 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:07.159 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:07.159 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:07.160 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:07.160 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:07.160 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:07.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:07.161 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:07.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:07.162 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:07.162 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:07.162 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:07.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:07.163 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:07Z" 2022-07-18T03:31:07.163 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:07.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:07Z" } 2022-07-18T03:31:07.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:07.166 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:07.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:07.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:12.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:12.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:12.169 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:12.225 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:12.226 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:12.308 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:12.309 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:12.309 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:12.309 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:12.310 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:12.310 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:12.310 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:12.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:12.311 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:12.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:12.311 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:12.311 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:12.312 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:12.312 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:12.312 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:12.312 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:12.313 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:12.313 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:12.313 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:12.314 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:12.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:12.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:12.314 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:12.315 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:12.315 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:12.315 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:12.315 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:12.316 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:12.316 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:12.316 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:12.316 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:12.317 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:12.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:12.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:12.317 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:12.318 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:12.318 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:12.318 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:12.318 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:12.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:12.319 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:12.319 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:12.320 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:12.320 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:12.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:12.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:12.321 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:12.321 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:12.321 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:12.321 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:12.322 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:12.322 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:12.322 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:12.322 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:12.322 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:12.323 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:12.323 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:12.323 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:12.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:12.324 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:12.324 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:12.324 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:12.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:12.325 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:12Z" 2022-07-18T03:31:12.325 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:12.326 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:12Z" } 2022-07-18T03:31:12.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:12.327 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:12.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:12.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:17.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:17.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:17.331 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:17.380 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:17.381 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:17.445 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:17.445 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:17.446 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:17.446 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:17.446 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:17.447 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:17.447 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:17.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:17.448 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:17.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:17.448 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:17.449 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:17.449 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:17.449 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:17.449 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:17.450 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:17.450 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:17.450 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:17.451 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:17.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:17.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:17.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:17.452 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:17.452 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:17.452 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:17.453 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:17.453 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:17.453 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:17.454 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:17.454 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:17.454 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:17.455 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:17.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:17.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:17.455 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:17.456 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:17.456 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:17.456 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:17.457 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:17.457 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:17.457 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:17.458 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:17.458 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:17.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:17.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:17.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:17.459 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:17.459 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:17.459 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:17.460 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:17.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:17.460 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:17.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:17.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:17.461 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:17.461 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:17.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:17.462 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:17.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:17.463 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:17.463 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:17.463 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:17.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:17.464 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:17Z" 2022-07-18T03:31:17.464 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:17.466 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:17Z" } 2022-07-18T03:31:17.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:17.467 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:17.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:17.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:22.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:22.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:22.471 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:22.526 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:22.526 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:22.644 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:22.644 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:22.644 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:22.645 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:22.645 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:22.645 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:22.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:22.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:22.646 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:22.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:22.646 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:22.647 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:22.647 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:22.647 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:22.647 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:22.648 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:22.648 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:22.648 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:22.649 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:22.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:22.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:22.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:22.650 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:22.650 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:22.650 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:22.651 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:22.651 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:22.651 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:22.651 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:22.651 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:22.652 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:22.652 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:22.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:22.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:22.653 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:22.653 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:22.653 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:22.654 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:22.654 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:22.654 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:22.654 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:22.655 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:22.655 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:22.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:22.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:22.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:22.656 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:22.656 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:22.657 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:22.657 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:22.657 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:22.658 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:22.658 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:22.658 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:22.658 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:22.659 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:22.659 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:22.659 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:22.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:22.660 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:22.660 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:22.660 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:22.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:22.661 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:22Z" 2022-07-18T03:31:22.661 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:22.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:22Z" } 2022-07-18T03:31:22.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:22.663 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:22.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:22.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:27.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:27.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:27.667 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:27.716 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:27.716 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:27.797 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:27.798 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:27.798 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:27.799 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:27.799 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:27.799 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:27.799 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:27.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:27.800 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:27.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:27.800 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:27.801 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:27.801 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:27.801 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:27.802 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:27.802 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:27.802 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:27.802 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:27.803 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:27.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:27.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:27.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:27.804 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:27.804 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:27.804 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:27.804 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:27.805 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:27.805 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:27.805 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:27.806 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:27.806 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:27.806 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:27.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:27.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:27.807 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:27.807 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:27.807 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:27.808 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:27.808 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:27.808 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:27.808 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:27.809 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:27.809 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:27.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:27.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:27.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:27.810 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:27.810 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:27.810 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:27.811 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:27.811 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:27.811 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:27.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:27.812 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:27.812 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:27.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:27.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:27.813 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:27.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:27.813 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:27.814 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:27.814 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:27.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:27.814 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:27Z" 2022-07-18T03:31:27.815 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:27.816 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:27Z" } 2022-07-18T03:31:27.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:27.817 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:27.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:27.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:32.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:32.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:32.821 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:32.877 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:32.877 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:32.950 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:32.951 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:32.951 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:32.951 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:32.952 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:32.952 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:32.952 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:32.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:32.953 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:32.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:32.953 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:32.953 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:32.954 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:32.954 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:32.954 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:32.954 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:32.955 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:32.955 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:32.955 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:32.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:32.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:32.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:32.957 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:32.957 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:32.957 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:32.957 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:32.958 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:32.958 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:32.958 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:32.958 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:32.959 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:32.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:32.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:32.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:32.960 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:32.960 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:32.960 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:32.960 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:32.961 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:32.961 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:32.961 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:32.961 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:32.961 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:32.962 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:32.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:32.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:32.963 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:32.963 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:32.963 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:32.964 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:32.964 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:32.965 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:32.965 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:32.965 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:32.966 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:32.966 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:32.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:32.967 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:32.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:32.968 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:32.968 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:32.968 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:32.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:32.969 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:32Z" 2022-07-18T03:31:32.969 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:32.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:32Z" } 2022-07-18T03:31:32.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:32.972 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:32.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:32.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:37.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:37.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:37.976 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:38.031 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:38.031 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:38.093 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:38.094 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:38.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:38.096 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:38.097 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:38.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:38.099 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:38.100 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:38.101 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:38.101 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:38.101 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:38.101 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:38.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:38.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:38.102 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:38.102 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:38.103 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:38.103 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:38.103 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:38.103 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:38.104 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:38.104 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:38.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:38.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:38.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:38.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:38.105 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:38.105 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:38.105 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:38.106 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:38.106 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:38.106 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:38.106 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:38.107 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:38.107 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:38.107 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:38.107 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:38.108 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:38.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:38.108 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:38.108 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:38.109 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:38.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:38.109 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:38Z" 2022-07-18T03:31:38.109 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:38.111 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:38Z" } 2022-07-18T03:31:38.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:38.112 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:38.112 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:38.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:43.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:43.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:43.116 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:43.166 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:43.166 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:43.229 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:43.229 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:43.230 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:43.230 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:43.230 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:43.231 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:43.231 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:43.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:43.231 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:43.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:43.232 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:43.232 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:43.233 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:43.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:43.233 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:43.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:43.234 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:43.235 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:43.235 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:43.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:43.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:43.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:43.236 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:43.236 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:43.237 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:43.237 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:43.237 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:43.238 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:43.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:43.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:43.241 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:43.242 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:43.243 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:43.244 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:43.245 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:43.246 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:43.247 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:43.248 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:43.249 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:43.250 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:43Z" 2022-07-18T03:31:43.251 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:43.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:43Z" } 2022-07-18T03:31:43.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:43.253 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:43.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:43.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:48.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:48.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:48.257 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:48.307 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:48.308 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:48.367 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:48.368 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:48.368 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:48.368 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:48.368 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:48.369 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:48.369 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:48.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:48.369 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:48.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:48.370 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:48.370 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:48.370 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:48.371 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:48.371 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:48.371 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:48.371 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:48.372 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:48.372 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:48.372 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:48.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:48.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:48.373 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:48.373 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:48.373 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:48.374 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:48.374 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:48.374 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:48.374 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:48.375 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:48.375 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:48.375 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:48.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:48.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:48.376 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:48.376 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:48.376 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:48.377 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:48.377 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:48.377 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:48.377 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:48.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:48.379 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:48.380 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:48.381 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:48.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:48.382 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:48.382 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:48.382 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:48.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:48.383 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:48Z" 2022-07-18T03:31:48.383 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:48.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:48Z" } 2022-07-18T03:31:48.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:48.386 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:48.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:48.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:53.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:53.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:53.389 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:53.455 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:53.456 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:53.562 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:53.563 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:53.563 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:53.563 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:53.564 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:53.564 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:53.564 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:53.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:53.565 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:53.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:53.565 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:53.565 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:53.566 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:53.566 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:53.566 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:53.567 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:53.567 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:53.567 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:53.568 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:53.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:53.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:53.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:53.569 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:53.569 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:53.569 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:53.569 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:53.570 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:53.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:53.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:53.571 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:53.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:53.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:53.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:53.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:53.572 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:53.572 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:53.572 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:53.573 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:53.573 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:53.573 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:53.574 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:53.574 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:53.574 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:53.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:53.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:53.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:53.575 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:53.576 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:53.576 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:53.576 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:53.576 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:53.577 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:53.577 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:53.577 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:53.578 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:53.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:53.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:53.578 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:53.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:53.579 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:53.579 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:53.580 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:53.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:53.580 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:53Z" 2022-07-18T03:31:53.580 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:53.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:53Z" } 2022-07-18T03:31:53.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:53.583 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:53.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:53.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:31:58.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:58.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:31:58.587 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:31:58.641 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:31:58.641 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:31:58.731 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:31:58.731 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:31:58.732 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:31:58.732 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:31:58.733 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:31:58.733 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:31:58.733 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:58.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:58.734 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:31:58.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:58.734 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:31:58.735 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:58.735 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:58.735 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:31:58.735 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:58.736 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:31:58.736 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:58.736 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:31:58.736 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:58.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:58.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:58.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:58.737 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:31:58.738 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:58.738 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:58.738 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:31:58.738 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:58.739 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:58.739 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:58.739 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:31:58.739 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:58.740 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:58.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:58.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:58.740 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:31:58.741 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:58.741 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:58.741 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:31:58.741 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:58.742 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:58.742 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:58.743 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:31:58.743 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:58.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:58.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:58.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:31:58.745 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:31:58.745 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:31:58.746 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:31:58.746 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:31:58.746 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:31:58.747 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:31:58.747 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:31:58.747 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:31:58.747 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:31:58.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:58.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:31:58.748 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:31:58.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:58.749 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:31:58.749 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:31:58.749 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:31:58.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:31:58.750 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:31:58Z" 2022-07-18T03:31:58.750 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:31:58.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:31:58Z" } 2022-07-18T03:31:58.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:31:58.754 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:31:58.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:31:58.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:03.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:03.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:03.759 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:03.815 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:03.815 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:03.889 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:03.889 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:03.890 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:03.890 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:03.890 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:03.891 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:03.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:03.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:03.891 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:03.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:03.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:03.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:03.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:03.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:03.893 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:03.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:03.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:03.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:03.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:03.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:03.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:03.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:03.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:03.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:03.896 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:03.897 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:03.897 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:03.897 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:03.897 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:03.898 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:03.898 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:03.898 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:03.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:03.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:03.899 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:03.899 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:03.899 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:03.900 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:03.900 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:03.900 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:03.901 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:03.901 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:03.902 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:03.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:03.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:03.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:03.903 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:03.903 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:03.903 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:03.904 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:03.904 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:03.904 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:03.904 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:03.904 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:03.905 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:03.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:03.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:03.906 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:03.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:03.907 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:03.907 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:03.907 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:03.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:03.908 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:03Z" 2022-07-18T03:32:03.908 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:03.910 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:03Z" } 2022-07-18T03:32:03.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:03.911 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:03.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:03.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:08.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:08.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:08.915 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:08.967 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:08.967 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:09.026 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:09.027 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:09.027 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:09.028 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:09.028 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:09.029 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:09.029 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:09.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:09.030 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:09.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:09.031 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:09.031 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:09.032 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:09.032 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:09.033 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:09.033 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:09.034 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:09.034 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:09.034 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:09.034 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:09.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:09.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:09.035 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:09.035 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:09.036 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:09.036 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:09.036 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:09.036 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:09.037 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:09.037 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:09.037 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:09.038 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:09.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:09.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:09.038 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:09.039 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:09.039 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:09.039 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:09.039 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:09.039 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:09.040 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:09.040 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:09.040 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:09.040 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:09.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:09.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:09.041 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:09.041 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:09.042 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:09.042 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:09.042 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:09.042 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:09.043 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:09.043 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:09.043 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:09.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:09.044 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:09.044 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:09.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:09.045 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:09.045 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:09.045 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:09.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:09.046 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:09Z" 2022-07-18T03:32:09.046 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:09.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:09Z" } 2022-07-18T03:32:09.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:09.049 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:09.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:09.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:14.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:14.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:14.053 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:14.110 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:14.110 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:14.188 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:14.188 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:14.189 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:14.189 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:14.189 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:14.189 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:14.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:14.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:14.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:14.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:14.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:14.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:14.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:14.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:14.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:14.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:14.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:14.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:14.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:14.194 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:14.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:14.195 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:14.195 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:14.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:14.196 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:14.196 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:14.196 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:14.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:14.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:14.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:14.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:14.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:14.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:14.199 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:14.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:14.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:14.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:14.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:14.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:14.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:14.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:14.201 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:14.202 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:14.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:14.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:14.203 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:14.203 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:14.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:14.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:14.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:14.204 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:14.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:14.204 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:14.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:14.205 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:14.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:14.205 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:14Z" 2022-07-18T03:32:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:14.207 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:14Z" } 2022-07-18T03:32:14.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:14.208 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:14.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:14.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:19.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:19.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:19.212 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:19.281 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:19.281 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:19.350 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:19.350 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:19.351 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:19.351 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:19.351 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:19.351 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:19.352 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:19.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:19.352 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:19.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:19.353 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:19.353 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:19.353 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:19.353 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:19.354 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:19.354 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:19.354 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:19.354 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:19.355 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:19.355 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:19.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:19.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:19.356 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:19.356 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:19.356 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:19.357 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:19.357 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:19.357 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:19.358 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:19.358 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:19.358 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:19.358 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:19.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:19.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:19.359 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:19.359 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:19.359 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:19.360 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:19.360 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:19.360 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:19.360 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:19.361 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:19.361 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:19.361 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:19.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:19.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:19.362 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:19.362 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:19.362 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:19.363 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:19.363 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:19.363 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:19.363 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:19.364 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:19.364 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:19.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:19.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:19.365 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:19.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:19.365 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:19.365 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:19.366 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:19.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:19.366 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:19Z" 2022-07-18T03:32:19.366 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:19.367 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:19Z" } 2022-07-18T03:32:19.367 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:19.368 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:19.368 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:19.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:24.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:24.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:24.373 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:24.428 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:24.428 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:24.511 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:24.511 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:24.512 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:24.512 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:24.512 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:24.513 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:24.513 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:24.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:24.514 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:24.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:24.514 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:24.515 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:24.515 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:24.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:24.516 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:24.516 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:24.517 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:24.517 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:24.517 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:24.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:24.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:24.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:24.518 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:24.518 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:24.519 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:24.519 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:24.519 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:24.519 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:24.520 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:24.520 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:24.520 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:24.520 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:24.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:24.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:24.521 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:24.521 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:24.522 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:24.522 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:24.522 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:24.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:24.523 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:24.523 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:24.523 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:24.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:24.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:24.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:24.525 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:24.525 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:24.526 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:24.526 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:24.526 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:24.526 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:24.527 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:24.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:24.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:24.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:24.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:24.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:24.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:24.528 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:24.529 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:24.529 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:24.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:24.529 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:24Z" 2022-07-18T03:32:24.530 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:24.531 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:24Z" } 2022-07-18T03:32:24.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:24.532 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:24.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:24.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:29.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:29.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:29.536 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:29.592 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:29.593 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:29.674 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:29.674 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:29.674 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:29.675 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:29.675 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:29.675 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:29.676 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:29.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:29.676 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:29.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:29.677 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:29.677 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:29.677 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:29.677 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:29.678 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:29.678 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:29.678 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:29.678 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:29.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:29.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:29.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:29.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:29.680 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:29.680 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:29.681 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:29.681 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:29.682 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:29.683 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:29.683 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:29.684 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:29.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:29.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:29.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:29.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:29.687 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:29.687 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:29.688 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:29.688 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:29.690 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:29.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:29.690 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:29.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:29.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:29.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:29.692 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:29.693 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:29.693 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:29.693 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:29.694 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:29.694 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:29.694 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:29.695 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:29.695 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:29.695 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:29.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:29.696 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:29.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:29.696 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:29.697 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:29.697 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:29.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:29.697 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:29Z" 2022-07-18T03:32:29.698 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:29.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:29Z" } 2022-07-18T03:32:29.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:29.700 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:29.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:29.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:34.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:34.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:34.705 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:34.758 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:34.758 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:34.843 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:34.844 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:34.844 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:34.844 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:34.845 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:34.845 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:34.845 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:34.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:34.846 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:34.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:34.846 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:34.847 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:34.847 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:34.847 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:34.848 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:34.848 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:34.848 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:34.849 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:34.849 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:34.849 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:34.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:34.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:34.850 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:34.851 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:34.851 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:34.851 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:34.851 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:34.852 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:34.852 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:34.852 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:34.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:34.853 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:34.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:34.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:34.854 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:34.854 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:34.855 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:34.855 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658114850.215685_140.1", 2022-07-18T03:32:34.856 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:34.856 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:34.857 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:34.857 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:27:30.215685Z", 2022-07-18T03:32:34.858 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:34.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:34.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:34.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:34.859 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:34.859 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:34.859 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:34.860 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:34.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:34.860 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:34.860 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:34.861 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:34.861 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:34.861 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:34.861 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:34.862 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:34.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:34.862 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:34.862 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:34.863 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:34.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:34.863 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:34Z" 2022-07-18T03:32:34.863 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:34.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658114850.215685_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:27:30.215685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:34Z" } 2022-07-18T03:32:34.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658114850.215685_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:27:30.215685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:34.866 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:34.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658114850.215685_140.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:34.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:39.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... FAIL 2022-07-18T03:32:39.888 DEBUG:boto:path=/mlvank-20/ 2022-07-18T03:32:39.888 DEBUG:boto:auth_path=/mlvank-20/ 2022-07-18T03:32:39.889 DEBUG:boto:Method: PUT 2022-07-18T03:32:39.889 DEBUG:boto:Path: /mlvank-20/ 2022-07-18T03:32:39.890 DEBUG:boto:Data: 2022-07-18T03:32:39.890 DEBUG:boto:Headers: {} 2022-07-18T03:32:39.891 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:32:39.892 DEBUG:boto:Port: 8000 2022-07-18T03:32:39.892 DEBUG:boto:Params: {} 2022-07-18T03:32:39.894 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:32:39.895 DEBUG:boto:Token: None 2022-07-18T03:32:39.896 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:32:39 GMT /mlvank-20/ 2022-07-18T03:32:39.898 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:ah5oPnlXb/Ag+TkcWpdt09Tth78= 2022-07-18T03:32:39.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:32:39 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:ah5oPnlXb/Ag+TkcWpdt09Tth78=', 'Content-Length': '0'} 2022-07-18T03:32:39.921 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001dbd8c7417aa9755-0062d4d457-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:32:39 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:32:39.923 DEBUG:rgw_multi.tests:created bucket=mlvank-20 2022-07-18T03:32:39.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:39.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:39.924 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:39.978 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:39.978 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:40.030 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:32:40.043 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:32:40.044 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:32:40.044 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:40.044 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:32:40.044 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:32:40.045 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:32:40.045 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:32:40.045 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:32:40.046 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:32:40.046 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:32:40.046 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:32:40.047 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:32:39.918685Z" 2022-07-18T03:32:40.047 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:32:40.047 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:32:40.047 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:32:40.048 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:32:40.048 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:32:40.048 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:32:40.049 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:40.050 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:32:40.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:40.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:40.052 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:40.114 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:40.115 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:40.223 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:40.224 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:40.224 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:40.224 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:40.225 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:40.225 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:40.225 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:40.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:40.226 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:40.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:40.227 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:40.227 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:40.227 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:40.228 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:40.228 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:40.228 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:40.228 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:40.229 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:40.229 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:40.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:40.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:40.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:40.230 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:40.230 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:40.231 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:40.231 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:40.231 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:40.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:40.232 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:40.232 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:40.232 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:40.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:40.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:40.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:40.233 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:40.234 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:40.234 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:40.235 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.911279_141.1", 2022-07-18T03:32:40.235 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:40.236 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:40.236 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:40.236 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.911279Z", 2022-07-18T03:32:40.236 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:40.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:40.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:40.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:40.238 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:40.238 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:40.238 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:40.238 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:40.239 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:40.239 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:40.239 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:40.239 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:40.240 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:40.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:40.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:40.241 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:40.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:40.241 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:40.241 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:40.242 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:40.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:40.242 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:40Z" 2022-07-18T03:32:40.242 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:40.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.911279_141.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.911279Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:40Z" } 2022-07-18T03:32:40.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.911279_141.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.911279Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:40.245 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:40.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.911279_141.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:40.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658115159.918685_144.1 target=1_1658115159.911279_141.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:45.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:45.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:45.249 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:45.299 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:45.300 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:45.382 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:45.382 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:45.383 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:45.383 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:45.384 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:45.384 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:45.384 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:45.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:45.385 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:45.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:45.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:45.386 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:45.386 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:45.387 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:45.387 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:45.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:45.387 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:45.388 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:45.388 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:45.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:45.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:45.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:45.389 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:45.389 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:45.390 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:45.390 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:45.391 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:45.391 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:45.392 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:45.392 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:45.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:45.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:45.394 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:32:45.395 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:32:45.396 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:45.397 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:45.398 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:45.399 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:45.400 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:45.400 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:45.401 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:45.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:45.401 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:45.402 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:45.402 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:45.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:45.403 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:45Z" 2022-07-18T03:32:45.403 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:45.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:45Z" } 2022-07-18T03:32:45.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:45.407 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:45.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:45.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:50.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:50.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:50.412 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:50.468 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:50.468 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:50.531 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:50.532 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:50.532 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:50.532 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:50.533 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:50.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:50.534 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:50.534 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:50.535 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:50.535 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:50.535 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:50.535 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:50.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:50.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:50.537 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:50.537 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:50.537 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:50.538 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:50.538 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:50.538 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:50.538 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:50.539 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:50.539 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:50.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:50.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:50.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:32:50.541 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:50.541 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:50.541 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:50.541 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:32:50.542 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:50.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:50.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:50.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:50.543 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:50.543 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:50.543 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:50.543 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:50.544 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:50.544 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:50.544 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:50.544 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:50.545 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:50.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:50.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:50.545 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:50.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:50.546 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:50.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:50.546 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:50.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:50.547 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:50Z" 2022-07-18T03:32:50.547 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:50.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:50Z" } 2022-07-18T03:32:50.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:50.549 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:50.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:50.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:32:55.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:55.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:32:55.553 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:32:55.608 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:32:55.608 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:32:55.699 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:32:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:32:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:32:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:32:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:32:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:32:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:32:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:32:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:32:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:32:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:32:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:32:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:32:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:32:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:32:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:32:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:32:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:32:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:32:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:32:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:32:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:32:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:32:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:32:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:32:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:32:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:32:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:32:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:32:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:32:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:32:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:32:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:32:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:32:55Z" 2022-07-18T03:32:55.716 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:32:55.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:32:55Z" } 2022-07-18T03:32:55.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:32:55.719 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:32:55.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:32:55.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:00.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:00.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:00.722 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:00.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:00.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:00.848 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:00.849 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:00.849 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:00.849 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:00.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:00.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:00.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:00.852 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:00.853 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:00.853 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:00.854 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:00.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:00.855 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:00.855 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:00.856 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:00.856 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:00.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:00.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:00.858 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:00.858 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:00.859 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:00.859 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:00.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:00.860 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:00.861 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:00.861 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:00.862 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:00.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:00.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:00.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:00.863 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:00.864 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:00.864 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:00.865 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:00.865 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:00.865 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:00.866 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:00.866 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:00.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:00.867 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:00.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:00.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:00.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:00.869 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:00.869 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:00.870 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:00.870 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:00.871 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:00.871 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:00.872 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:00.872 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:00.872 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:00.873 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:00.873 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:00.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:00.874 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:00.874 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:00.875 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:00.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:00.875 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:00Z" 2022-07-18T03:33:00.876 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:00.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:00Z" } 2022-07-18T03:33:00.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:00.878 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:00.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:00.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:05.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:05.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:05.883 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:05.946 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:05.946 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:06.017 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:06Z" 2022-07-18T03:33:06.033 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:06.034 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:06Z" } 2022-07-18T03:33:06.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:06.036 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:06.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:06.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:11.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:11.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:11.039 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:11.088 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:11.089 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:11.158 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:11.159 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:11.159 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:11.160 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:11.160 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:11.161 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:11.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:11.161 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:11.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:11.162 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:11.162 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:11.163 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:11.163 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:11.163 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:11.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:11.164 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:11.164 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:11.164 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:11.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:11.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:11.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:11.165 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:11.166 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:11.166 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:11.166 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:11.166 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:11.168 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:11.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:11.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:11.168 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:11.169 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:11.169 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:11.169 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:11.170 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:11.170 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:11.170 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:11.170 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:11.171 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:11.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:11.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:11.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:11.172 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:11.172 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:11.172 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:11.172 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:11.173 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:11.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:11.173 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:11.173 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:11.174 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:11.174 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:11.174 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:11.174 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:11.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:11.175 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:11.175 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:11.175 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:11.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:11.176 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:11Z" 2022-07-18T03:33:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:11.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:11Z" } 2022-07-18T03:33:11.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:11.179 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:11.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:11.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:16.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:16.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:16.182 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:16.238 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:16.239 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:16.317 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:16.318 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:16.318 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:16.319 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:16.319 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:16.319 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:16.319 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:16.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:16.321 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:16.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:16.322 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:16.322 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:16.323 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:16.323 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:16.324 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:16.324 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:16.325 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:16.326 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:16.326 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:16.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:16.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:16.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:16.328 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:16.329 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:16.329 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:16.330 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:16.330 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:16.330 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:16.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:16.331 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:16.331 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:16.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:16.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:16.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:16.332 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:16.333 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:16.333 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:16.333 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:16.333 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:16.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:16.334 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:16.334 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:16.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:16.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:16.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:16.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:16.335 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:16.336 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:16.336 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:16.336 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:16.337 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:16.337 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:16.338 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:16.338 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:16.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:16.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:16.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:16.341 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:16.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:16.342 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:16.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:16.343 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:16.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:16.344 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:16Z" 2022-07-18T03:33:16.344 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:16.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:16Z" } 2022-07-18T03:33:16.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:16.349 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:16.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:16.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:21.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:21.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:21.356 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:21.412 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:21.413 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:21.475 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:21.476 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:21.476 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:21.477 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:21.477 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:21.477 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:21.478 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:21.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:21.478 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:21.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:21.479 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:21.479 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:21.479 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:21.479 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:21.480 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:21.480 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:21.480 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:21.480 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:21.481 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:21.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:21.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:21.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:21.482 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:21.482 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:21.482 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:21.482 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:21.484 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:21.484 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:21.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:21.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:21.485 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:21.485 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:21.485 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:21.485 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:21.486 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:21.486 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:21.486 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:21.487 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:21.487 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:21.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:21.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:21.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:21.488 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:21.488 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:21.488 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:21.488 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:21.489 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:21.489 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:21.489 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:21.489 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:21.490 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:21.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:21.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:21.491 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:21.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:21.491 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:21.491 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:21.492 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:21.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:21.492 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:21Z" 2022-07-18T03:33:21.492 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:21.494 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:21Z" } 2022-07-18T03:33:21.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:21.495 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:21.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:21.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:26.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:26.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:26.499 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:26.554 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:26.555 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:26.641 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:26.641 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:26.642 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:26.642 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:26.642 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:26.642 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:26.643 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:26.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:26.643 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:26.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:26.644 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:26.644 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:26.644 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:26.645 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:26.645 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:26.645 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:26.645 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:26.646 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:26.646 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:26.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:26.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:26.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:26.647 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:26.647 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:26.648 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:26.648 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:26.648 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:26.648 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:26.649 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:26.649 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:26.649 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:26.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:26.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:26.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:26.650 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:26.651 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:26.651 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:26.651 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:26.651 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:26.652 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:26.652 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:26.652 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:26.652 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:26.653 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:26.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:26.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:26.653 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:26.654 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:26.654 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:26.654 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:26.654 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:26.655 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:26.655 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:26.655 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:26.655 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:26.656 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:26.656 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:26.656 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:26.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:26.657 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:26.657 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:26.657 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:26.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:26.658 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:26Z" 2022-07-18T03:33:26.658 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:26.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:26Z" } 2022-07-18T03:33:26.660 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:26.660 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:26.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:26.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:31.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:31.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:31.664 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:31.733 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:31.734 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:31.796 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:31.796 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:31.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:31.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:31.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:31.804 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:31.805 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:31.805 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:31.806 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:31.806 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:31.807 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:31.812 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:31.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:31.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:31.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:31.815 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:31Z" 2022-07-18T03:33:31.820 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:31.822 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:31Z" } 2022-07-18T03:33:31.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:31.823 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:31.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:31.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:36.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:36.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:36.827 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:36.883 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:36.883 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:36.952 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:36.952 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:36.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:36.961 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:36.961 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:36.961 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:36.961 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:36.962 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:36.962 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:36.962 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:36.963 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:36.963 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:36.963 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:36.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:36.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:36.964 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:36.964 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:36.964 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:36.965 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:36.965 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:36.965 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:36.965 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:36.966 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:36.966 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:36.966 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:36.966 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:36.967 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:36.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:36.967 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:36.967 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:36.968 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:36.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:36.968 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:36Z" 2022-07-18T03:33:36.968 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:36.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:36Z" } 2022-07-18T03:33:36.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:36.971 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:36.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:36.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:41.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:41.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:41.974 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:42.030 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:42.030 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:42.100 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:42.100 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:42.100 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:42.101 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:42.101 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:42.101 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:42.107 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:42.107 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:42.107 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:42.107 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:42.108 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:42.108 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:42.108 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:42.108 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:42.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:42.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:42.109 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:42.110 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:42.110 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:42.110 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:42.111 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:42.111 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:42.111 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:42.111 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:42.112 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:42.112 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:42.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:42.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:42.113 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:42.113 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:42.113 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:42.114 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:42.114 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:42.114 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:42.114 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:42.115 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:42.115 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:42.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:42.116 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:42.116 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:42.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:42.116 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:42.117 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:42.117 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:42.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:42.117 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:42Z" 2022-07-18T03:33:42.118 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:42.119 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:42Z" } 2022-07-18T03:33:42.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:42.121 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:42.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:42.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:47.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:47.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:47.125 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:47.181 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:47.181 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:47.253 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:47.254 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:47.254 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:47.254 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:47.258 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:47.258 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:47.258 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:47.263 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:47.263 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:47.263 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:47.264 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:47.264 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:47.264 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:47.265 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:47.265 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:47.265 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:47.265 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:47.267 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:47.267 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:47.268 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:47.268 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:47.269 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:47.270 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:47.270 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:47.270 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:47.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:47.271 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:47.271 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:47.271 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:47.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:47.272 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:47Z" 2022-07-18T03:33:47.272 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:47.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:47Z" } 2022-07-18T03:33:47.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:47.275 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:47.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:47.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:52.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:52.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:52.279 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:52.330 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:52.331 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:52.396 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:52.397 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:52.397 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:52.398 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:52.398 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:52.399 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:52.399 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:52.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:52.400 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:52.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:52.401 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:52.401 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:52.401 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:52.402 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:52.402 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:52.403 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:52.403 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:52.403 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:52.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:52.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:52.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:52.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:52.406 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:52.406 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:52.406 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:52.407 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:52.407 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:52.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:52.408 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:52.409 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:52.409 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:52.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:52.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:52.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:52.411 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:52.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:52.412 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:52.412 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:52.413 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:52.413 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:52.413 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:52.414 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:52.415 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:52.415 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:52.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:52.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:52.416 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:52.417 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:52.417 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:52.417 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:52.418 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:52.418 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:52.419 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:52.419 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:52.420 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:52.420 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:52.420 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:52.420 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:52.422 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:52.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:52.422 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:52Z" 2022-07-18T03:33:52.422 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:52.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:52Z" } 2022-07-18T03:33:52.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:52.425 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:52.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:52.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:33:57.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:57.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:33:57.429 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:33:57.483 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:33:57.484 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:33:57.567 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:33:57.567 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:33:57.567 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:33:57.568 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:33:57.568 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:33:57.568 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:33:57.569 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:57.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:57.569 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:33:57.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:57.569 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:33:57.570 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:57.570 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:57.570 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:33:57.571 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:57.571 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:33:57.571 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:57.571 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:33:57.572 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:57.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:57.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:57.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:57.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:33:57.573 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:57.573 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:57.573 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:33:57.574 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:57.574 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:57.574 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:57.574 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:33:57.575 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:57.575 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:57.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:57.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:57.576 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:33:57.576 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:57.577 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:57.577 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:33:57.577 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:57.577 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:57.578 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:57.578 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:33:57.578 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:57.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:33:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:33:57.579 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:33:57.580 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:33:57.580 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:33:57.580 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:33:57.580 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:33:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:33:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:33:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:33:57.581 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:57.582 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:33:57.582 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:33:57.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:33:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:33:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:33:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:33:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:33:57Z" 2022-07-18T03:33:57.584 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:33:57.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:33:57Z" } 2022-07-18T03:33:57.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:33:57.586 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:33:57.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:33:57.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:02.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:02.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:02.593 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:02.649 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:02.649 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:02.714 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:02.717 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:02.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:02.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:02.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:02.718 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:02.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:02.718 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:02.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:02.719 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:02.719 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:02.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:02.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:02.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:02.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:02.720 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:02.721 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:02.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:02.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:02.724 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:02.724 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:02.724 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:02.725 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:02.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:02.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:02.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:02Z" 2022-07-18T03:34:02.731 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:02.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:02Z" } 2022-07-18T03:34:02.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:02.734 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:02.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:02.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:07.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:07.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:07.739 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:07.795 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:07.796 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:07.890 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:07.901 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:07.901 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:07.901 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:07.903 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:07.903 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:07.903 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:07.903 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:07.904 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:07.904 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:07.904 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:07.904 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:07.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:07.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:07.905 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:07.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:07.906 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:07.906 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:07.906 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:07.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:07.907 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:07Z" 2022-07-18T03:34:07.907 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:07.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:07Z" } 2022-07-18T03:34:07.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:07.910 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:07.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:07.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:12.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:12.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:12.913 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:12.964 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:12.965 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:13.033 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:13.039 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:13.039 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:13.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:13.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:13.040 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:13.040 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:13.040 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:13.040 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:13.041 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:13.042 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:13.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:13.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:13.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:13.043 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:13.043 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:13.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:13.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:13.045 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:13.045 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:13.046 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:13.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:13.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:13.048 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:13.048 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:13.049 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:13.049 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:13.050 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:13.050 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:13.051 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:13.051 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:13.052 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:13.052 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:13.053 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:13.053 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:13.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:13Z" 2022-07-18T03:34:13.057 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:13.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:13Z" } 2022-07-18T03:34:13.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:13.061 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:13.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:13.063 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:18.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:18.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:18.067 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:18.119 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:18.119 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:18.187 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:18.187 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:18.188 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:18.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:18.190 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:18.191 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:18.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:18.193 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:18.194 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:18.194 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:18.194 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:18.194 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:18.195 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:18.195 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:18.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:18.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:18.196 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:18.196 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:18.196 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:18.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:18.197 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:18.197 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:18.197 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:18.197 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:18.198 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:18.198 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:18.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:18.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:18.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:18.200 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:18.200 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:18.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:18.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:18.203 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:18Z" 2022-07-18T03:34:18.203 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:18.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:18Z" } 2022-07-18T03:34:18.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:18.205 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:18.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:18.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:23.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:23.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:23.209 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:23.265 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:23.266 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:23.337 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:23.338 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:23.338 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:23.338 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:23.339 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:23.339 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:23.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:23.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:23.340 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:23.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:23.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:23.341 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:23.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:23.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:23.341 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:23.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:23.342 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:23.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:23.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:23.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:23.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:23.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:23.343 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:23.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:23.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:23.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:23.345 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:23.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:23.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:23.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:23.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:23.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:23.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:23.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:23.347 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:23.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:23.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:23.347 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:23.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:23.348 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:23.348 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:23.348 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:23.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:23.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:23.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:23.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:23.350 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:23.350 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:23.350 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:23.350 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:23.352 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:23.353 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:23.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:23.354 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:23Z" 2022-07-18T03:34:23.354 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:23.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:23Z" } 2022-07-18T03:34:23.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:23.357 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:23.357 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:23.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:28.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:28.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:28.360 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:28.414 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:28.415 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:28.491 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:28.492 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:28.492 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:28.492 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:28.492 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:28.493 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:28.493 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:28.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:28.493 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:28.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:28.494 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:28.494 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:28.494 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:28.495 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:28.495 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:28.495 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:28.495 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:28.496 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:28.496 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:28.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:28.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:28.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:28.497 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:28.497 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:28.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:28.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:28.498 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:28.498 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:28.499 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:28.499 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:28.499 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:28.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:28.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:28.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:28.500 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:28.500 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:28.501 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:28.501 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:28.501 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:28.501 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:28.502 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:28.502 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:28.502 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:28.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:28.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:28.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:28.504 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:28.504 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:28.504 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:28.504 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:28.505 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:28.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:28.505 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:28.505 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:28.506 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:28.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:28.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:28.506 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:28.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:28.507 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:28.507 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:28.507 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:28.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:28.508 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:28Z" 2022-07-18T03:34:28.508 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:28.509 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:28Z" } 2022-07-18T03:34:28.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:28.511 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:28.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:28.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:33.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:33.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:33.515 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:33.563 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:33.564 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:33.641 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:33.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:33.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:33.651 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:33.651 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:33.652 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:33.652 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:33.652 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:33.652 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:33.653 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:33.653 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:33.653 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:33.653 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:33.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:33.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:33.654 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:33.654 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:33.655 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:33.655 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:33.656 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:33.656 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:33.656 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:33.656 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:33.657 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:33.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:33.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:33.657 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:33.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:33.658 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:33.658 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:33.658 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:33.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:33.659 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:33Z" 2022-07-18T03:34:33.659 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:33.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:33Z" } 2022-07-18T03:34:33.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:33.662 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:33.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:33.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:38.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:38.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:38.666 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:38.715 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:38.716 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:38.804 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:38.805 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:38.805 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:38.805 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:38.805 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:38.806 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:38.806 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:38.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:38.807 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:38.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:38.807 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:38.807 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:38.808 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:38.808 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:38.808 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:38.808 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:38.809 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:38.809 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:38.809 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:38.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:38.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:38.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:38.810 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:38.811 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:38.811 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:38.811 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:38.811 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:38.812 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:38.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:38.812 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:38.812 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:38.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:38.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:38.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:38.814 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:38.814 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:38.814 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:38.814 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:38.815 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:38.815 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:38.815 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:38.815 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:38.816 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:38.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:38.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:38.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:38.817 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:38.817 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:38.817 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:38.817 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:38.818 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:38.818 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:38.818 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:38.818 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:38.819 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:38.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:38.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:38.819 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:38.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:38.820 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:38.820 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:38.820 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:38.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:38.821 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:38Z" 2022-07-18T03:34:38.821 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:38.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:38Z" } 2022-07-18T03:34:38.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:38.824 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:38.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:38.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:43.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:43.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:43.828 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:43.884 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:43.884 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:43.950 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:43.950 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:43.950 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:43.951 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:43.951 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:43.951 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:43.952 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:43.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:43.952 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:43.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:43.953 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:43.953 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:43.953 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:43.953 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:43.954 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:43.954 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:43.954 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:43.955 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:43.955 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:43.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:43.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:43.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:43.956 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:43.956 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:43.956 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:43.957 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:43.957 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:43.957 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:43.957 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:43.958 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:43.958 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:43.958 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:43.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:43.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:43.959 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:43.959 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:43.959 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:43.960 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:43.960 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:43.960 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:43.960 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:43.961 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:43.961 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:43.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:43.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:43.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:43.962 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:43.962 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:43.962 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:43.963 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:43.963 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:43.963 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:43.963 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:43.964 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:43.964 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:43.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:43.964 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:43.965 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:43.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:43.965 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:43.965 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:43.966 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:43.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:43.966 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:43Z" 2022-07-18T03:34:43.966 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:43.968 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:43Z" } 2022-07-18T03:34:43.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:43.969 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:43.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:43.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:48.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:48.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:48.972 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:49.028 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:49.029 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:49.111 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:49.111 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:49.112 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:49.112 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:49.112 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:49.112 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:49.113 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:49.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:49.113 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:49.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:49.114 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:49.114 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:49.114 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:49.114 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:49.115 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:49.115 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:49.115 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:49.115 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:49.116 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:49.116 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:49.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:49.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:49.117 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:49.117 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:49.117 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:49.117 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:49.118 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:49.118 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:49.118 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:49.118 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:49.119 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:49.119 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:49.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:49.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:49.120 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:49.120 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:49.120 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:49.120 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:49.121 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:49.121 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:49.121 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:49.121 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:49.122 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:49.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:49.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:49.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:49.123 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:49.123 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:49.123 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:49.123 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:49.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:49.124 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:49.124 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:49.124 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:49.125 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:49.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:49.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:49.125 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:49.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:49.126 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:49.126 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:49.126 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:49.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:49.127 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:49Z" 2022-07-18T03:34:49.127 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:49.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:49Z" } 2022-07-18T03:34:49.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:49.130 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:49.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:49.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:54.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:54.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:54.133 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:54.187 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:54.187 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:54.255 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:54.256 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:54.256 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:54.256 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:54.257 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:54.257 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:54.257 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:54.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:54.258 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:54.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:54.258 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:54.259 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:54.259 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:54.259 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:54.259 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:54.260 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:54.260 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:54.260 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:54.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:54.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:54.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:54.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:54.261 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:54.262 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:54.262 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:54.262 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:54.263 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:54.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:54.263 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:54.263 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:54.263 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:54.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:54.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:54.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:54.264 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:54.265 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:54.265 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:54.265 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:54.265 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:54.266 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:54.266 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:54.266 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:54.267 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:54.267 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:54.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:54.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:54.268 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:54.268 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:54.269 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:54.269 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:54.269 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:54.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:54.270 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:54.270 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:54.271 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:54.271 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:54.271 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:54.271 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:54.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:54.272 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:54.272 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:54.273 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:54.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:54.273 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:54Z" 2022-07-18T03:34:54.274 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:54.275 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:54Z" } 2022-07-18T03:34:54.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:54.277 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:54.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:54.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:34:59.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:59.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:34:59.280 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:34:59.328 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:34:59.329 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:34:59.408 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:34:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:34:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:34:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:34:59.410 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:34:59.410 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:34:59.410 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:59.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:59.411 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:34:59.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:59.411 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:34:59.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:59.412 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:59.412 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:34:59.412 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:59.413 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:34:59.413 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:59.413 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:34:59.414 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:59.414 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:59.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:59.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:59.415 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:34:59.415 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:34:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:59.417 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:59.417 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:34:59.417 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:59.417 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:59.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:59.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:59.418 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:34:59.419 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:59.419 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:59.419 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:34:59.419 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:59.420 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:59.420 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:59.420 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:34:59.420 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:59.421 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:59.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:59.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:34:59.421 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:34:59.422 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:34:59.422 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:34:59.422 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:34:59.422 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:34:59.423 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:34:59.423 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:34:59.423 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:34:59.423 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:34:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:34:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:34:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:34:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:34:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:34:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:34:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:34:59Z" 2022-07-18T03:34:59.427 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:34:59.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:34:59Z" } 2022-07-18T03:34:59.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:34:59.430 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:34:59.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:34:59.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:04.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:04.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:04.433 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:04.486 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:04.487 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:04.562 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:04.562 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:04.563 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:04.563 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:04.563 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:04.564 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:04.564 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:04.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:04.565 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:04.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:04.566 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:04.566 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:04.567 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:04.567 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:04.567 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:04.568 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:04.568 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:04.578 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:04.578 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:04.578 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:04.578 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:04Z" 2022-07-18T03:35:04.581 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:04.583 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:04Z" } 2022-07-18T03:35:04.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:04.584 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:04.585 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:04.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:09.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:09.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:09.588 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:09.645 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:09.646 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:09.712 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:09.713 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:09.713 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:09.714 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:09.715 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:09.716 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:09.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:09.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:09.717 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:09.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:09.718 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:09.719 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:09.719 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:09.720 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:09.720 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:09.720 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:09.721 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:09.721 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:09.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:09.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:09.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:09.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:09.724 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:09.724 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:09.725 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:09.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:09.726 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:09.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:09.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:09.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:09.728 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:09.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:09.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:09.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:09.730 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:09.732 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:09.732 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:09.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:09.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:09.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:09.735 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:09.737 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:09.737 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:09.738 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:09.738 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:09.739 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:09.739 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:09.740 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:09.740 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:09.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:09.740 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:09.741 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:09.741 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:09.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:09.741 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:09Z" 2022-07-18T03:35:09.742 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:09.743 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:09Z" } 2022-07-18T03:35:09.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:09.744 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:09.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:09.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:14.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:14.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:14.748 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:14.804 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:14.804 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:14.868 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:14.868 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:14.869 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:14.869 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:14.869 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:14.869 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:14.870 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:14.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:14.870 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:14.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:14.871 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:14.871 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:14.871 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:14.871 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:14.872 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:14.872 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:14.872 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:14.873 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:14.873 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:14.873 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:14.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:14.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:14.874 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:14.874 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:14.874 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:14.875 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:14.875 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:14.875 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:14.875 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:14.876 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:14.876 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:14.876 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:14.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:14.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:14.877 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:14.877 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:14.877 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:14.878 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:14.878 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:14.878 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:14.879 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:14.879 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:14.879 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:14.879 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:14.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:14.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:14.880 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:14.880 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:14.881 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:14.881 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:14.881 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:14.881 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:14.882 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:14.882 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:14.882 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:14.882 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:14.883 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:14.883 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:14.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:14.883 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:14.884 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:14.884 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:14.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:14.884 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:14Z" 2022-07-18T03:35:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:14.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:14Z" } 2022-07-18T03:35:14.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:14.887 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:14.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:14.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:19.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:19.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:19.891 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:19.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:19.948 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:20.033 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:20.033 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:20.034 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:20.034 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:20.034 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:20.036 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:20.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:20.036 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:20.041 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:20.041 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:20.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:20.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:20.044 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:20.044 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:20.045 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:20.045 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:20.046 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:20.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:20.047 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:20.048 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:20.048 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:20.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:20.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:20.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:20.049 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:20.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:20.050 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:20.050 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:20.050 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:20.050 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:20.051 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:20.051 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:20.051 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:20.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:20Z" 2022-07-18T03:35:20.054 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:20.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:20Z" } 2022-07-18T03:35:20.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:20.057 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:20.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:20.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:25.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:25.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:25.061 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:25.117 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:25.117 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:25.197 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:25.198 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:25.199 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:25.199 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:25.200 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:25.200 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:25.201 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:25.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:25.202 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:25.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:25.203 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:25.204 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:25.204 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:25.205 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:25.206 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:25.206 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:25.207 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:25.207 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:25.208 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:25.208 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:25.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:25.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:25.210 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:25.211 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:25.211 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:25.211 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:25.212 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:25.212 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:25.212 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:25.212 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:25.213 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:25.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:25.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:25.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:25.214 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:25.214 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:25.215 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:25.215 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:25.216 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:25.216 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:25.217 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:25.217 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:25.218 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:25.218 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:25.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:25.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:25.220 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:25.222 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:25.222 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:25.223 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:25.223 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:25.224 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:25.224 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:25.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:25.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:25.227 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:25.227 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:25.228 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:25.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:25.229 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:25Z" 2022-07-18T03:35:25.229 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:25.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:25Z" } 2022-07-18T03:35:25.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:25.234 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:25.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:25.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:30.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:30.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:30.240 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:30.395 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:30.405 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:30.405 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:30.405 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:30.405 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:30.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:30.406 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:30.406 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:30.406 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:30.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:30.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:30.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:30.407 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:30.408 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:30.408 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:30.408 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:30.408 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:30.409 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:30.409 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:30.409 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:30.409 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:30.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:30.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:30.410 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:30.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:30.411 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:30.411 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:30.411 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:30.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:30.412 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:30Z" 2022-07-18T03:35:30.412 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:30.413 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:30Z" } 2022-07-18T03:35:30.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:30.414 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:30.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:30.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:35.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:35.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:35.419 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:35.475 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:35.524 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:35.550 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:35.551 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:35.551 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:35.551 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:35.551 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:35.552 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:35.552 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:35.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:35.553 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:35.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:35.557 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:35.565 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:35.565 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:35.569 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:35.569 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:35.570 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:35.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:35.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:35.571 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:35.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:35.571 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:35.572 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:35.572 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:35.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:35.572 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:35Z" 2022-07-18T03:35:35.573 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:35.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:35Z" } 2022-07-18T03:35:35.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:35.576 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:35.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:35.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:40.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:40.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:40.581 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:40.636 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:40.637 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:40.719 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:40.720 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:40.721 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:40.721 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:40.722 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:40.722 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:40.723 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:40.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:40.724 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:40.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:40.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:40.725 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:40.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:40.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:40.727 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:40.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:40.728 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:40.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:40.728 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:40.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:40.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:40.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:40.729 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:40.730 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:40.730 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:40.730 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:40.730 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:40.731 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:40.731 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:40.731 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:40.732 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:40.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:40.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:40.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:40.733 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:40.733 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:40.733 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:40.734 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:40.734 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:40.734 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:40.735 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:40.735 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:40.735 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:40.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:40.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:40.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:40.736 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:40.736 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:40.737 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:40.737 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:40.737 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:40.737 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:40.738 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:40.738 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:40.738 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:40.739 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:40.739 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:40.739 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:40.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:40.739 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:40.740 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:40.740 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:40.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:40.741 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:40Z" 2022-07-18T03:35:40.741 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:40.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:40Z" } 2022-07-18T03:35:40.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:40.744 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:40.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:40.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:45.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:45.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:45.747 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:45.802 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:45.803 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:45.865 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:45.866 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:45.866 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:45.866 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:45.867 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:45.867 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:45.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:45.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:45.868 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:45.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:45.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:45.868 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:45.869 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:45.869 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:45.869 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:45.869 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:45.870 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:45.870 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:45.870 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:45.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:45.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:45.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:45.872 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:45.873 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:45.874 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:45.875 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:45.875 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:45.876 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:45.876 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:45.876 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:45.877 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:45.877 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:45.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:45.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:45.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:45.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:45.880 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:45.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:45.881 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:45.881 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:45.882 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:45.882 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:45.883 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:45.883 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:45.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:45.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:45.885 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:45.885 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:45.886 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:45.886 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:45.886 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:45.886 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:45.887 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:45.887 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:45.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:45.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:45.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:45.888 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:45.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:45.889 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:45.889 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:45.889 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:45.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:45.890 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:45Z" 2022-07-18T03:35:45.890 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:45.891 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:45Z" } 2022-07-18T03:35:45.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:45.893 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:45.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:45.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:50.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:50.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:50.898 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:50.954 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:50.955 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:51.037 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:51.037 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:51.037 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:51.038 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:51.038 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:51.038 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:51.038 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:51.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:51.039 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:51.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:51.039 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:51.040 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:51.040 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:51.040 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:51.040 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:51.041 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:51.041 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:51.041 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:51.041 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:51.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:51.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:51.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:51.043 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:51.043 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:51.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:51.043 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:51.044 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:51.044 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:51.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:51.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:51.045 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:51.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:51.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:51.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:51.046 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:51.046 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:51.046 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:51.046 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:51.047 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:51.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:51.047 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:51.047 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:51.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:51.053 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:51Z" 2022-07-18T03:35:51.053 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:51.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:51Z" } 2022-07-18T03:35:51.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:51.056 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:51.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:51.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:35:56.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:56.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:35:56.060 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:35:56.115 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:35:56.115 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:35:56.192 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:35:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:35:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:35:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:35:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:35:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:35:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:35:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:35:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:35:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:35:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:35:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:35:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:35:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:35:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:35:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:35:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:35:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:35:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:35:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:35:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:35:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:35:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:35:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:35:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:35:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:35:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:35:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:35:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:35:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:35:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:35:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:35:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:35:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:35:56Z" 2022-07-18T03:35:56.209 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:35:56.211 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:35:56Z" } 2022-07-18T03:35:56.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:35:56.212 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:35:56.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:35:56.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:01.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:01.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:01.216 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:01.271 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:01.271 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:01.333 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:01.333 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:01.344 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:01.344 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:01.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:01.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:01.345 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:01.345 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:01.345 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:01.345 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:01.346 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:01.346 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:01.346 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:01.346 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:01.347 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:01.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:01.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:01.347 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:01.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:01.348 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:01.348 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:01.348 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:01.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:01.349 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:01Z" 2022-07-18T03:36:01.349 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:01.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:01Z" } 2022-07-18T03:36:01.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:01.352 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:01.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:01.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:06.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:06.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:06.356 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:06.406 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:06.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:06.462 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:06.463 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:06.463 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:06.463 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:06.464 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:06.464 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:06.464 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:06.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:06.465 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:06.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:06.465 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:06.465 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:06.466 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:06.466 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:06.466 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:06.467 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:06.467 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:06.467 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:06.467 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:06.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:06.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:06.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:06.469 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:06.469 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:06.469 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:06.469 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:06.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:06.470 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:06.470 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:06.471 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:06.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:06.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:06.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:06.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:06.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:06.472 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:06.472 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:06.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:06.473 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:06.473 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:06.473 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:06.474 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:06.474 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:06.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:06.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:06.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:06.475 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:06.475 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:06.475 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:06.476 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:06.476 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:06.476 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:06.476 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:06.477 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:06.477 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:06.477 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:06.477 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:06.478 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:06.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:06.478 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:06.479 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:06.479 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:06.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:06.479 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:06Z" 2022-07-18T03:36:06.480 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:06.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:06Z" } 2022-07-18T03:36:06.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:06.482 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:06.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:06.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:11.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:11.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:11.486 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:11.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:11.543 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:11.626 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:11.627 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:11.627 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:11.627 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:11.627 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:11.628 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:11.628 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:11.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:11.628 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:11.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:11.629 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:11.629 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:11.629 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:11.630 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:11.630 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:11.630 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:11.630 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:11.631 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:11.631 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:11.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:11.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:11.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:11.632 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:11.632 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:11.633 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:11.633 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:11.633 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:11.633 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:11.634 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:11.634 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:11.634 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:11.634 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:11.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:11.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:11.635 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:11.635 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:11.636 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:11.636 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:11.636 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:11.636 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:11.637 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:11.637 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:11.637 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:11.638 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:11.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:11.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:11.638 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:11.639 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:11.639 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:11.639 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:11.639 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:11.640 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:11.640 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:11.640 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:11.640 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:11.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:11.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:11.641 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:11.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:11.642 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:11.642 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:11.642 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:11.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:11.643 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:11Z" 2022-07-18T03:36:11.643 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:11.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:11Z" } 2022-07-18T03:36:11.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:11.644 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:11.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:11.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:16.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:16.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:16.648 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:16.704 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:16.705 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:16.801 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:16.817 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:16.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:16.818 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:16.818 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:16.819 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:16.819 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:16.820 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:16.820 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:16.821 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:16.821 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:16.822 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:16.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:16.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:16.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:16.824 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:16.824 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:16.825 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:16.825 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:16.825 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:16.826 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:16.827 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:16.827 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:16.827 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:16.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:16.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:16.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:16.829 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:16.830 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:16.830 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:16.831 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:16.831 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:16.832 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:16.832 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:16.833 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:16.833 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:16.834 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:16.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:16.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:16.836 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:16.836 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:16.837 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:16.838 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:16.839 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:16.839 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:16.840 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:16.840 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:16.841 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:16.841 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:16.842 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:16.842 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:16.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:16.843 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:16.844 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:16.845 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:16.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:16.846 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:16Z" 2022-07-18T03:36:16.847 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:16.847 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:16Z" } 2022-07-18T03:36:16.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:16.849 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:16.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:16.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:21.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:21.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:21.854 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:21.915 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:21.917 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:21.976 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:21.976 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:21.976 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:21.977 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:21.978 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:21.978 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:21.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:21.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:21.981 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:21.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:21.984 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:21.984 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:21.985 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:21.985 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:21.986 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:21.986 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:21.987 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:21.987 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:21.988 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:21.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:21.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:21.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:21.990 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:21.990 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:21.991 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:21.992 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:21.994 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:21.994 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:21.995 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:21.995 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:21.996 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:21.996 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:21.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:21.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:21.998 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:21.998 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:21.998 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:21.999 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:21.999 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:22.000 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:22.000 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:22.001 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:22.001 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:22.002 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:22.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:22.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:22.005 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:22.005 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:22.006 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:22.006 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:22.007 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:22.007 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:22.008 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:22.008 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:22.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:22.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:22.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:22.011 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:22.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:22.012 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:22.012 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:22.013 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:22.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:22.014 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:21Z" 2022-07-18T03:36:22.014 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:22.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:21Z" } 2022-07-18T03:36:22.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:22.018 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:22.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:22.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:27.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:27.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:27.023 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:27.072 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:27.073 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:27.142 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:27.142 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:27.142 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:27.143 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:27.143 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:27.143 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:27.143 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:27.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:27.145 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:27.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:27.145 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:27.146 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:27.146 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:27.146 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:27.147 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:27.147 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:27.147 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:27.147 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:27.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:27.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:27.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:27.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:27.149 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:27.149 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:27.150 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:27.150 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:27.150 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:27.151 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:27.151 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:27.152 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:27.152 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:27.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:27.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:27.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:27.155 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:27.155 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:27.156 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:27.156 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:27.156 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:27.156 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:27.157 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:27.157 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:27.157 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:27.158 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:27.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:27.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:27.159 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:27.159 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:27.160 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:27.160 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:27.160 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:27.160 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:27.161 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:27.161 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:27.161 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:27.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:27.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:27.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:27.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:27.163 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:27.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:27.163 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:27.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:27.164 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:27Z" 2022-07-18T03:36:27.164 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:27.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:27Z" } 2022-07-18T03:36:27.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:27.167 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:27.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:27.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:32.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:32.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:32.173 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:32.230 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:32.231 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:32.318 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:32.319 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:32.319 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:32.320 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:32.320 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:32.320 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:32.321 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:32.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:32.321 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:32.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:32.322 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:32.323 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:32.323 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:32.324 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:32.324 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:32.326 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:32.326 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:32.327 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:32.327 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:32.328 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:32.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:32.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:32.330 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:32.330 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:32.331 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:32.331 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:32.332 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:32.332 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:32.333 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:32.333 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:32.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:32.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:32.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:32.338 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:32.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:32.340 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:32.340 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:32.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:32.343 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:32.344 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:32.345 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:32.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:32.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:32.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:32.347 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:32.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:32.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:32.352 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:32.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:32.353 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:32Z" 2022-07-18T03:36:32.353 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:32.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:32Z" } 2022-07-18T03:36:32.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:32.357 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:32.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:32.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:37.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:37.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:37.362 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:37.417 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:37.424 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:37.485 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:37.486 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:37.486 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:37.487 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:37.487 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:37.487 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:37.487 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:37.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:37.488 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:37.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:37.488 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:37.489 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:37.489 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:37.489 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:37.489 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:37.490 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:37.490 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:37.490 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:37.491 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:37.491 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:37.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:37.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:37.492 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:37.492 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:37.492 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:37.492 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:37.493 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:37.493 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:37.493 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:37.493 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:37.494 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:37.494 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:37.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:37.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:37.495 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:37.495 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:37.495 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:37.495 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:37.496 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:37.496 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:37.496 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:37.496 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:37.497 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:37.497 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:37.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:37.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:37.498 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:37.498 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:37.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:37.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:37.499 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:37.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:37.499 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:37.499 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:37.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:37.502 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:37Z" 2022-07-18T03:36:37.502 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:37.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:37Z" } 2022-07-18T03:36:37.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:37.505 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:37.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:37.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:42.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:42.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:42.508 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:42.563 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:42.564 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:42.640 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:42Z" 2022-07-18T03:36:42.657 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:42.658 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:42Z" } 2022-07-18T03:36:42.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:42.660 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:42.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:42.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:47.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:47.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:47.665 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:47.728 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:47.728 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:47.820 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:47.820 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:47.821 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:47.821 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:47.821 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:47.821 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:47.822 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:47.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:47.822 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:47.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:47.823 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:47.823 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:47.823 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:47.825 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:47.825 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:47.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:47.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:47.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:47.826 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:47.826 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:47.826 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:47.828 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:47.828 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:47.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:47.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:47.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:47.829 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:47.829 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:47.829 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:47.830 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:47.830 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:47.830 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:47.831 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:47.831 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:47.831 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:47.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:47.833 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:47.833 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:47.833 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:47.833 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:47.834 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:47.834 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:47.834 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:47.836 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:47.836 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:47.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:47Z" 2022-07-18T03:36:47.837 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:47.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:47Z" } 2022-07-18T03:36:47.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:47.839 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:47.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:47.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:52.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:52.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:52.843 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:52.899 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:52.900 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:52.973 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:52.973 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:52.974 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:52.974 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:52.974 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:52.974 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:52.975 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:52.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:52.975 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:52.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:52.976 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:52.976 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:52.976 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:52.976 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:52.977 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:52.977 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:52.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:52.978 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:52.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:52.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:52.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:52.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:52.979 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:52.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:52.980 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:52.980 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:52.980 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:52.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:52.981 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:52.981 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:52.981 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:52.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:52.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:52.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:52.982 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:52.983 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:52.983 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:52.983 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:52.983 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:52.984 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:52.984 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:52.984 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:52.984 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:52.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:52.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:52.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:52.985 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:52.986 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:52.986 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:52.986 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:52.987 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:52.987 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:52.987 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:52.988 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:52.988 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:52.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:52.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:52.989 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:52.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:52.989 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:52.989 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:52.990 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:52.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:52.990 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:52Z" 2022-07-18T03:36:52.991 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:52.992 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:52Z" } 2022-07-18T03:36:52.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:52.993 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:52.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:52.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:36:57.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:57.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:36:57.997 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:36:58.053 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:36:58.054 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:36:58.113 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:36:58.113 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:36:58.114 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:36:58.114 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:36:58.114 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:36:58.115 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:36:58.115 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:58.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:58.115 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:36:58.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:58.116 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:36:58.116 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:58.117 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:58.117 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:36:58.117 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:58.117 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:36:58.118 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:58.118 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:36:58.118 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:58.119 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:58.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:58.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:58.120 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:36:58.120 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:58.120 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:58.121 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:36:58.121 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:58.121 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:58.122 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:58.122 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:36:58.122 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:58.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:58.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:58.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:58.123 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:36:58.124 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:58.124 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:58.124 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:36:58.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:58.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:58.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:58.125 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:36:58.126 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:58.126 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:58.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:58.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:36:58.127 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:36:58.127 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:36:58.127 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:36:58.128 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:36:58.128 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:36:58.128 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:36:58.128 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:36:58.129 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:36:58.129 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:36:58.129 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:58.129 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:36:58.130 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:36:58.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:58.130 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:36:58.131 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:36:58.131 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:36:58.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:36:58.131 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:36:58Z" 2022-07-18T03:36:58.132 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:36:58.133 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:36:58Z" } 2022-07-18T03:36:58.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:36:58.134 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:36:58.135 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:36:58.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:03.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:03.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:03.138 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:03.194 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:03.195 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:03.271 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:03.272 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:03.273 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:03.273 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:03.273 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:03.274 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:03.274 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:03.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:03.274 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:03.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:03.275 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:03.275 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:03.275 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:03.276 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:03.276 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:03.276 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:03.276 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:03.277 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:03.277 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:03.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:03.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:03.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:03.278 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:03.278 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:03.278 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:03.278 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:03.279 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:03.279 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:03.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:03.280 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:03.280 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:03.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:03.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:03.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:03.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:03.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:03.282 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:03.282 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:03.282 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:03.282 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:03.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:03.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:03.284 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:03.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:03.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:03.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:03.285 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:03.285 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:03.285 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:03.285 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:03.286 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:03.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:03.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:03.286 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:03.287 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:03.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:03.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:03.288 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:03.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:03.288 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:03.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:03.289 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:03.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:03.290 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:03Z" 2022-07-18T03:37:03.290 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:03.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:03Z" } 2022-07-18T03:37:03.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:03.292 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:03.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:03.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:08.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:08.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:08.296 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:08.352 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:08.353 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:08.422 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:08.423 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:08.423 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:08.423 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:08.424 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:08.424 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:08.424 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:08.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:08.425 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:08.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:08.425 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:08.425 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:08.426 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:08.426 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:08.426 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:08.426 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:08.427 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:08.427 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:08.427 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:08.428 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:08.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:08.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:08.429 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:08.429 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:08.429 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:08.429 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:08.430 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:08.430 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:08.430 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:08.430 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:08.431 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:08.431 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:08.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:08.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:08.432 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:08.432 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:08.432 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:08.432 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:08.433 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:08.433 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:08.433 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:08.433 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:08.434 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:08.434 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:08.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:08.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:08.435 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:08.435 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:08.435 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:08.436 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:08.436 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:08.436 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:08.436 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:08.437 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:08.437 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:08.437 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:08.437 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:08.438 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:08.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:08.438 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:08.438 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:08.439 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:08.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:08.439 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:08Z" 2022-07-18T03:37:08.439 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:08.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:08Z" } 2022-07-18T03:37:08.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:08.442 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:08.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:08.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:13.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:13.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:13.447 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:13.501 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:13.502 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:13.577 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:13.578 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:13.578 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:13.579 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:13.579 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:13.579 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:13.580 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:13.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:13.580 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:13.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:13.581 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:13.581 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:13.582 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:13.582 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:13.582 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:13.583 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:13.583 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:13.584 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:13.584 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:13.585 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:13.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:13.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:13.586 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:13.587 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:13.587 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:13.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:13.588 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:13.588 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:13.589 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:13.589 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:13.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:13.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:13.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:13.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:13.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:13.591 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:13.592 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:13.592 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:13.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:13.592 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:13.593 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:13.593 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:13.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:13.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:13.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:13.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:13.595 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:13.595 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:13.595 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:13.595 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:13.596 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:13.596 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:13.596 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:13.597 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:13.597 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:13.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:13.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:13.599 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:13.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:13.599 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:13.600 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:13.600 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:13.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:13.601 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:13Z" 2022-07-18T03:37:13.601 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:13.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:13Z" } 2022-07-18T03:37:13.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:13.604 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:13.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:13.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:18.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:18.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:18.608 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:18.664 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:18.665 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:18.734 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:18.734 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:18.735 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:18.735 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:18.735 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:18.735 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:18.736 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:18.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:18.736 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:18.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:18.737 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:18.737 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:18.737 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:18.737 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:18.738 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:18.738 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:18.738 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:18.738 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:18.739 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:18.739 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:18.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:18.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:18.740 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:18.740 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:18.740 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:18.741 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:18.741 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:18.742 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:18.742 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:18.742 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:18.742 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:18.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:18.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:18.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:18.743 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:18.744 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:18.744 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:18.744 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:18.744 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:18.745 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:18.745 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:18.745 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:18.745 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:18.746 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:18.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:18.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:18.746 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:18.747 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:18.747 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:18.747 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:18.747 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:18.748 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:18.748 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:18.748 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:18.748 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:18.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:18.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:18.749 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:18.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:18.750 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:18.750 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:18.750 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:18.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:18.751 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:18Z" 2022-07-18T03:37:18.751 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:18.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:18Z" } 2022-07-18T03:37:18.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:18.754 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:18.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:18.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:23.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:23.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:23.758 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:23.813 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:23.814 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:23.886 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:23Z" 2022-07-18T03:37:23.913 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:23.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:23Z" } 2022-07-18T03:37:23.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:23.916 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:23.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:23.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:28.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:28.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:28.920 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:28.978 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:28.979 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:29.070 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:29Z" 2022-07-18T03:37:29.088 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:29.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:29Z" } 2022-07-18T03:37:29.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:29.090 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:29.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:29.091 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:34.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:34.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:34.094 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:34.143 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:34.144 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:34.211 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:34.212 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:34.212 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:34.213 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:34.213 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:34.214 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:34.214 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:34.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:34.215 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:34.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:34.216 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:34.216 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:34.217 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:34.217 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:34.218 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:34.218 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:34.218 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:34.218 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:34.219 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:34.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:34.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:34.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:34.220 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:34.220 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:34.220 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:34.220 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:34.221 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:34.221 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:34.221 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:34.221 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:34.222 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:34.222 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:34.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:34.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:34.223 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:34.223 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:34.223 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:34.223 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:34.224 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:34.224 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:34.224 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:34.224 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:34.225 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:34.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:34.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:34.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:34.226 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:34.226 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:34.226 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:34.226 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:34.227 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:34.227 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:34.227 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:34.228 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:34.228 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:34.228 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:34.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:34.229 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:34.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:34.229 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:34.230 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:34.230 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:34.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:34.231 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:34Z" 2022-07-18T03:37:34.231 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:34.235 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:34Z" } 2022-07-18T03:37:34.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:34.237 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:34.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:34.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:39.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:39.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:39.243 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:39.300 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:39.300 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:39.364 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:39.365 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:39.365 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:39.365 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:39.366 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:39.366 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:39.366 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:39.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:39.367 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:39.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:39.367 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:39.367 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:39.368 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:39.368 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:39.368 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:39.368 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:39.369 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:39.369 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:39.369 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:39.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:39.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:39.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:39.371 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:39.371 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:39.371 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:39.371 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:39.372 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:39.372 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:39.372 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:39.373 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:39.373 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:39.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:39.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:39.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:39.375 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:39.375 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:39.375 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:39.376 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:39.376 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:39.376 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:39.376 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:39.377 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:39.377 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:39.377 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:39.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:39.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:39.378 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:39.378 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:39.379 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:39.379 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:39.379 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:39.379 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:39.380 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:39.380 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:39.380 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:39.380 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:39.381 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:39.381 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:39.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:39.381 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:39.382 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:39.382 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:39.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:39.382 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:39Z" 2022-07-18T03:37:39.383 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:39.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:39Z" } 2022-07-18T03:37:39.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:39.386 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:39.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:39.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:44.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:44.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:44.392 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:44.448 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:44.449 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:44.534 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:44.535 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:44.536 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:44.536 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:44.537 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:44.538 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:44.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:44.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:44.539 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:44.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:44.542 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:44.543 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:44.543 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:44.544 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:44.544 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:44.545 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:44.546 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:44.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:44.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:44.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:44.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:44.549 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:44.549 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:44.550 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:44.551 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:44.551 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:44.552 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:44.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:44.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:44.554 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:44.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:44.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:44.556 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:44.556 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:44.557 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:44.558 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115159.918685_144.1", 2022-07-18T03:37:44.558 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:44.558 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:44.559 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:44.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:32:39.918685Z", 2022-07-18T03:37:44.560 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:44.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:44.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:44.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:44.562 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:44.563 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:44.563 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:44.564 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:44.564 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:44.565 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:44.566 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:44.566 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:44.567 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:44.567 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:44.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:44.569 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:44.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:44.570 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:44.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:44.571 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:44.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:44.572 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:44Z" 2022-07-18T03:37:44.573 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:44.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115159.918685_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:32:39.918685Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:44Z" } 2022-07-18T03:37:44.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115159.918685_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:32:39.918685Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:44.577 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:44.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115159.918685_144.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:44.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:49.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2022-07-18T03:37:49.586 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-21 2022-07-18T03:37:49.587 DEBUG:boto:path=/mlvank-21/ 2022-07-18T03:37:49.588 DEBUG:boto:auth_path=/mlvank-21/ 2022-07-18T03:37:49.588 DEBUG:boto:Method: PUT 2022-07-18T03:37:49.589 DEBUG:boto:Path: /mlvank-21/ 2022-07-18T03:37:49.589 DEBUG:boto:Data: 2022-07-18T03:37:49.590 DEBUG:boto:Headers: {} 2022-07-18T03:37:49.590 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:37:49.591 DEBUG:boto:Port: 8000 2022-07-18T03:37:49.591 DEBUG:boto:Params: {} 2022-07-18T03:37:49.593 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:37:49.593 DEBUG:boto:Token: None 2022-07-18T03:37:49.594 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:37:49 GMT /mlvank-21/ 2022-07-18T03:37:49.595 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:MiL0VpKdZgKaurzpEtcT0l/nhe4= 2022-07-18T03:37:49.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:37:49 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:MiL0VpKdZgKaurzpEtcT0l/nhe4=', 'Content-Length': '0'} 2022-07-18T03:37:49.618 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000088b3c82b8368fa2-0062d4d58d-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:37:49 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:37:49.619 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-22 2022-07-18T03:37:49.620 DEBUG:boto:path=/mlvank-22/ 2022-07-18T03:37:49.620 DEBUG:boto:auth_path=/mlvank-22/ 2022-07-18T03:37:49.621 DEBUG:boto:Method: PUT 2022-07-18T03:37:49.621 DEBUG:boto:Path: /mlvank-22/ 2022-07-18T03:37:49.622 DEBUG:boto:Data: 2022-07-18T03:37:49.623 DEBUG:boto:Headers: {} 2022-07-18T03:37:49.623 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:37:49.624 DEBUG:boto:Port: 8000 2022-07-18T03:37:49.624 DEBUG:boto:Params: {} 2022-07-18T03:37:49.625 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:37:49.625 DEBUG:boto:Token: None 2022-07-18T03:37:49.626 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:37:49 GMT /mlvank-22/ 2022-07-18T03:37:49.627 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:yqbzZoSsdd6mzM3/e2vIUt6GbGA= 2022-07-18T03:37:49.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:37:49 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:yqbzZoSsdd6mzM3/e2vIUt6GbGA=', 'Content-Length': '0'} 2022-07-18T03:37:49.669 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a05bbc344bd5b4b4-0062d4d58d-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:37:49 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:37:49.671 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-23 2022-07-18T03:37:49.671 DEBUG:boto:path=/mlvank-23/ 2022-07-18T03:37:49.672 DEBUG:boto:auth_path=/mlvank-23/ 2022-07-18T03:37:49.673 DEBUG:boto:Method: PUT 2022-07-18T03:37:49.674 DEBUG:boto:Path: /mlvank-23/ 2022-07-18T03:37:49.674 DEBUG:boto:Data: 2022-07-18T03:37:49.675 DEBUG:boto:Headers: {} 2022-07-18T03:37:49.675 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:37:49.676 DEBUG:boto:Port: 8001 2022-07-18T03:37:49.676 DEBUG:boto:Params: {} 2022-07-18T03:37:49.678 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:37:49.679 DEBUG:boto:Token: None 2022-07-18T03:37:49.680 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:37:49 GMT /mlvank-23/ 2022-07-18T03:37:49.681 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:VTBwcyGfSof9L8EJPU11VWf+RLw= 2022-07-18T03:37:49.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:37:49 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:VTBwcyGfSof9L8EJPU11VWf+RLw=', 'Content-Length': '0'} 2022-07-18T03:37:49.788 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f54b6830fe20fee8-0062d4d58d-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:37:49 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:37:49.789 DEBUG:boto:path=/mlvank-21/ 2022-07-18T03:37:49.789 DEBUG:boto:auth_path=/mlvank-21/ 2022-07-18T03:37:49.790 DEBUG:boto:path=/mlvank-21/?versioning 2022-07-18T03:37:49.790 DEBUG:boto:auth_path=/mlvank-21/?versioning 2022-07-18T03:37:49.791 DEBUG:boto:Method: PUT 2022-07-18T03:37:49.791 DEBUG:boto:Path: /mlvank-21/?versioning 2022-07-18T03:37:49.792 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:37:49.792 DEBUG:boto:Headers: {} 2022-07-18T03:37:49.793 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:37:49.793 DEBUG:boto:Port: 8000 2022-07-18T03:37:49.794 DEBUG:boto:Params: {} 2022-07-18T03:37:49.794 DEBUG:boto:Token: None 2022-07-18T03:37:49.795 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:37:49 GMT /mlvank-21/?versioning 2022-07-18T03:37:49.796 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:KGGfgGsjRwYRhGaKJnP81drH8yY= 2022-07-18T03:37:49.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:37:49 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:KGGfgGsjRwYRhGaKJnP81drH8yY=', 'Content-Length': '228'} 2022-07-18T03:37:49.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e07f9800afadea0e-0062d4d58d-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:37:49 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:37:49.804 DEBUG:boto:path=/mlvank-22/ 2022-07-18T03:37:49.805 DEBUG:boto:auth_path=/mlvank-22/ 2022-07-18T03:37:49.805 DEBUG:boto:path=/mlvank-22/?versioning 2022-07-18T03:37:49.806 DEBUG:boto:auth_path=/mlvank-22/?versioning 2022-07-18T03:37:49.806 DEBUG:boto:Method: PUT 2022-07-18T03:37:49.807 DEBUG:boto:Path: /mlvank-22/?versioning 2022-07-18T03:37:49.808 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:37:49.808 DEBUG:boto:Headers: {} 2022-07-18T03:37:49.809 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:37:49.810 DEBUG:boto:Port: 8000 2022-07-18T03:37:49.811 DEBUG:boto:Params: {} 2022-07-18T03:37:49.812 DEBUG:boto:Token: None 2022-07-18T03:37:49.813 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:37:49 GMT /mlvank-22/?versioning 2022-07-18T03:37:49.814 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:09Mv0R1Zf7Ol51Qy6wDjSguRgLw= 2022-07-18T03:37:49.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:37:49 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:09Mv0R1Zf7Ol51Qy6wDjSguRgLw=', 'Content-Length': '228'} 2022-07-18T03:37:49.825 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b0082a811032080-0062d4d58d-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:37:49 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:37:49.826 DEBUG:boto:path=/mlvank-23/ 2022-07-18T03:37:49.826 DEBUG:boto:auth_path=/mlvank-23/ 2022-07-18T03:37:49.827 DEBUG:boto:path=/mlvank-23/?versioning 2022-07-18T03:37:49.827 DEBUG:boto:auth_path=/mlvank-23/?versioning 2022-07-18T03:37:49.828 DEBUG:boto:Method: PUT 2022-07-18T03:37:49.828 DEBUG:boto:Path: /mlvank-23/?versioning 2022-07-18T03:37:49.829 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:37:49.829 DEBUG:boto:Headers: {} 2022-07-18T03:37:49.830 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:37:49.830 DEBUG:boto:Port: 8001 2022-07-18T03:37:49.831 DEBUG:boto:Params: {} 2022-07-18T03:37:49.832 DEBUG:boto:Token: None 2022-07-18T03:37:49.832 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:37:49 GMT /mlvank-23/?versioning 2022-07-18T03:37:49.833 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Aty5qj77pW2qTfV/+78/zxzQSJk= 2022-07-18T03:37:49.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:37:49 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Aty5qj77pW2qTfV/+78/zxzQSJk=', 'Content-Length': '228'} 2022-07-18T03:37:49.845 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b31c09831f1ce41-0062d4d58d-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:37:49 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:37:49.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:49.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:49.848 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:49.909 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:49.910 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:49.961 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:37:49.975 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:37:49.976 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:37:49.976 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:49.977 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:37:49.977 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:37:49.977 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:37:49.977 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:37:49.978 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:37:49.978 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:37:49.979 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:37:49.979 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:37:49.979 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:37:49.839831Z" 2022-07-18T03:37:49.979 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:37:49.980 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:37:49.980 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:37:49.980 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:37:49.980 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:37:49.981 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:37:49.982 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:49.983 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:37:49.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:49.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:49.984 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:50.035 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:50.035 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:50.132 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:50.133 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:50.133 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:50.133 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:50.134 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:50.134 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:50.134 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:50.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:50.135 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:50.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:50.135 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:50.136 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:50.136 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:50.136 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:50.136 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:50.137 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:50.137 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:50.137 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:50.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:50.140 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:50.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:50.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:50.141 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:50.141 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:50.141 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:50.142 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:50.142 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:50.142 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:50.143 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:50.143 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:50.143 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:50.144 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:50.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:50.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:50.145 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:50.146 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:50.146 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:50.147 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.642800_153.1", 2022-07-18T03:37:50.147 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:50.147 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:50.148 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:50.148 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.642800Z", 2022-07-18T03:37:50.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:50.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:50.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:50.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:50.149 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:50.149 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:50.150 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:50.150 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:50.150 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:50.150 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:50.151 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:50.151 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:50.151 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:50.151 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:50.152 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:50.152 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:50.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:50.152 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:50.153 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:50.153 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:50.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:50.153 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:50Z" 2022-07-18T03:37:50.154 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:50.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.642800_153.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.642800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:50Z" } 2022-07-18T03:37:50.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.642800_153.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.642800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:50.156 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:50.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.642800_153.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:50.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658115469.839831_166.1 target=1_1658115469.642800_153.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:37:55.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:55.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:37:55.161 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:37:55.216 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:37:55.217 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:37:55.284 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:37:55.285 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:37:55.285 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:37:55.285 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:37:55.286 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:37:55.286 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:37:55.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:55.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:55.287 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:37:55.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:55.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:37:55.287 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:55.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:55.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:37:55.288 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:55.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:37:55.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:55.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:37:55.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:55.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:55.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:55.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:55.290 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:37:55.291 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:55.291 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:55.291 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:37:55.292 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:55.292 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:55.292 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:55.292 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:37:55.293 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:55.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:55.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:55.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:55.294 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:37:55.294 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:55.294 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:55.294 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:37:55.295 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:55.295 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:55.295 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:55.295 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:37:55.296 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:55.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:55.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:55.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:37:55.297 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:37:55.297 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:37:55.297 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:37:55.297 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:37:55.298 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:37:55.298 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:37:55.298 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:37:55.298 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:37:55.299 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:37:55.299 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:55.299 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:37:55.299 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:37:55.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:55.300 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:37:55.300 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:37:55.301 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:37:55.301 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:37:55.301 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:37:55Z" 2022-07-18T03:37:55.301 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:37:55.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:37:55Z" } 2022-07-18T03:37:55.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:37:55.304 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:37:55.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:37:55.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:00.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:00.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:00.307 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:00.386 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:00.386 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:00.522 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:00.522 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:00.523 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:00.523 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:00.523 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:00.524 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:00.524 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:00.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:00.525 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:00.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:00.525 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:00.525 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:00.526 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:00.526 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:00.526 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:00.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:00.527 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:00.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:00.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:00.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:00.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:00.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:00.529 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:00.529 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:00.529 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:00.530 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:00.530 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:00.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:00.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:00.531 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:00.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:00.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:00.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:00.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:00.532 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:00.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:00.532 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:00.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:00.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:00.533 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:00.533 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:00.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:00.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:00.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:00.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:00.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:00.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:00.535 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:00.536 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:00.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:00.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:00.536 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:00.537 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:00.537 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:00.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:00.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:00.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:00.538 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:00.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:00.539 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:00.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:00.539 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:00.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:00.540 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:00Z" 2022-07-18T03:38:00.540 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:00.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:00Z" } 2022-07-18T03:38:00.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:00.542 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:00.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:00.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:05.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:05.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:05.546 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:05.594 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:05.595 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:05.652 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:05.653 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:05.653 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:05.653 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:05.653 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:05.654 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:05.654 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:05.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:05.655 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:05.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:05.655 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:05.655 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:05.656 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:05.656 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:05.656 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:05.656 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:05.657 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:05.657 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:05.657 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:05.659 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:05.659 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:05.659 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:05.659 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:05.660 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:05.660 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:05.660 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:05.660 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:05.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:05.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:05.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:05.661 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:05.662 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:05.662 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:05.662 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:05.662 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:05.663 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:05.663 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:05.663 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:05.663 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:05.664 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:05.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:05.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:05.664 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:05.665 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:05.665 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:05.665 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:05.666 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:05.666 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:05.666 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:05.666 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:05.667 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:05.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:05.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:05.667 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:05.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:05.668 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:05.668 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:05.668 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:05.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:05.669 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:05Z" 2022-07-18T03:38:05.669 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:05.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:05Z" } 2022-07-18T03:38:05.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:05.672 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:05.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:05.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:10.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:10.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:10.675 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:10.727 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:10.727 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:10.791 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:10.792 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:10.792 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:10.793 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:10.793 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:10.794 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:10.794 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:10.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:10.795 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:10.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:10.796 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:10.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:10.796 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:10.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:10.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:10.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:10.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:10.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:10.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:10.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:10.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:10.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:10.799 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:10.800 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:10.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:10.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:10.800 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:10.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:10.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:10.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:10.802 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:10.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:10.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:10.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:10.803 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:10.803 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:10.803 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:10.803 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:10.804 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:10.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:10.804 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:10.805 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:10.805 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:10.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:10.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:10.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:10.806 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:10.806 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:10.806 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:10.807 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:10.807 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:10.807 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:10.807 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:10.808 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:10.808 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:10.808 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:10.808 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:10.809 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:10.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:10.809 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:10.809 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:10.810 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:10.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:10.810 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:10Z" 2022-07-18T03:38:10.810 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:10.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:10Z" } 2022-07-18T03:38:10.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:10.813 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:10.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:10.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:15.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:15.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:15.817 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:15.871 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:15.872 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:15.956 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:15.957 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:15.957 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:15.958 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:15.958 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:15.959 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:15.959 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:15.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:15.960 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:15.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:15.961 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:15.962 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:15.962 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:15.963 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:15.963 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:15.964 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:15.964 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:15.964 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:15.965 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:15.965 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:15.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:15.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:15.966 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:15.966 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:15.966 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:15.967 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:15.967 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:15.967 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:15.967 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:15.968 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:15.968 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:15.968 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:15.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:15.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:15.969 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:15.969 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:15.969 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:15.970 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:15.970 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:15.970 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:15.971 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:15.971 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:15.971 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:15.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:15.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:15.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:15.972 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:15.972 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:15.972 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:15.973 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:15.973 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:15.973 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:15.973 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:15.974 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:15.974 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:15.974 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:15.975 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:15.975 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:15.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:15.975 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:15.975 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:15.976 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:15.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:15.976 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:15Z" 2022-07-18T03:38:15.976 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:15.978 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:15Z" } 2022-07-18T03:38:15.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:15.979 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:15.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:15.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:20.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:20.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:20.984 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:21.040 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:21.041 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:21.102 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:21.103 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:21.103 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:21.103 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:21.104 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:21.104 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:21.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:21.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:21.105 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:21.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:21.105 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:21.105 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:21.106 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:21.106 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:21.106 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:21.106 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:21.107 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:21.107 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:21.107 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:21.108 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:21.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:21.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:21.109 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:21.109 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:21.109 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:21.109 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:21.109 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:21.110 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:21.110 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:21.110 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:21.110 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:21.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:21.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:21.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:21.112 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:21.112 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:21.113 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:21.113 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:21.113 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:21.113 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:21.114 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:21.114 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:21.114 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:21.114 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:21.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:21.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:21.115 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:21.116 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:21.116 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:21.116 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:21.116 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:21.117 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:21.117 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:21.117 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:21.118 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:21.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:21.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:21.118 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:21.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:21.119 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:21.119 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:21.119 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:21.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:21.120 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:21Z" 2022-07-18T03:38:21.120 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:21.121 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:21Z" } 2022-07-18T03:38:21.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:21.122 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:21.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:21.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:26.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:26.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:26.127 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:26.182 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:26.182 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:26.247 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:26.247 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:26.247 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:26.248 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:26.248 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:26.248 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:26.248 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:26.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:26.249 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:26.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:26.249 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:26.250 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:26.250 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:26.250 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:26.250 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:26.251 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:26.251 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:26.251 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:26.252 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:26.252 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:26.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:26.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:26.253 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:26.253 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:26.253 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:26.253 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:26.254 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:26.254 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:26.254 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:26.255 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:26.255 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:26.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:26.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:26.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:26.256 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:26.256 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:26.256 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:26.257 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:26.257 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:26.257 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:26.257 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:26.258 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:26.258 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:26.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:26.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:26.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:26.259 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:26.259 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:26.260 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:26.260 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:26.260 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:26.260 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:26.260 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:26.261 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:26.261 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:26.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:26.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:26.262 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:26.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:26.262 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:26.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:26.263 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:26.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:26.263 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:26Z" 2022-07-18T03:38:26.264 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:26.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:26Z" } 2022-07-18T03:38:26.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:26.266 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:26.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:26.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:31.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:31.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:31.270 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:31.328 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:31.328 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:31.392 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:31.393 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:31.393 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:31.393 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:31.394 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:31.394 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:31.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:31.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:31.395 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:31.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:31.395 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:31.395 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:31.396 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:31.396 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:31.396 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:31.396 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:31.397 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:31.397 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:31.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:31.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:31.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:31.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:31.398 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:31.399 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:31.399 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:31.399 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:31.399 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:31.400 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:31.400 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:31.400 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:31.400 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:31.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:31.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:31.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:31.401 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:31.402 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:31.402 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:31.402 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:31.403 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:31.403 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:31.403 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:31.403 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:31.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:31.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:31.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:31.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:31.405 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:31.405 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:31.405 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:31.405 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:31.406 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:31.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:31.406 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:31.406 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:31.407 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:31.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:31.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:31.407 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:31.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:31.408 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:31.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:31.409 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:31.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:31.409 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:31Z" 2022-07-18T03:38:31.410 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:31.411 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:31Z" } 2022-07-18T03:38:31.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:31.412 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:31.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:31.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:36.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:36.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:36.416 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:36.471 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:36.471 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:36.536 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:36Z" 2022-07-18T03:38:36.552 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:36.554 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:36Z" } 2022-07-18T03:38:36.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:36.555 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:36.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:36.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:41.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:41.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:41.559 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:41.665 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:41.665 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:41.666 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:41.666 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:41.666 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:41.666 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:41.667 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:41.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:41.667 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:41.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:41.668 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:41.668 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:41.668 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:41.668 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:41.669 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:41.669 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:41.669 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:41.669 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:41.670 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:41.670 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:41.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:41.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:41.671 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:41.671 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:41.671 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:41.671 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:41.672 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:41.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:41.672 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:41.672 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:41.673 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:41.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:41.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:41.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:41.674 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:41.674 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:41.674 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:41.674 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:41.675 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:41.675 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:41.675 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:41.675 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:41.676 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:41.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:41.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:41.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:41.677 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:41.677 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:41.677 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:41.677 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:41.678 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:41.678 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:41.678 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:41.679 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:41.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:41.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:41.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:41.680 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:41.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:41.680 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:41.680 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:41.681 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:41.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:41.681 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:41Z" 2022-07-18T03:38:41.681 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:41.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:41Z" } 2022-07-18T03:38:41.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:41.684 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:41.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:41.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:46.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:46.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:46.687 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:46.741 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:46.741 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:46.830 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:46.830 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:46.831 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:46.831 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:46.831 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:46.831 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:46.832 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:46.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:46.832 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:46.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:46.833 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:46.833 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:46.833 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:46.833 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:46.834 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:46.834 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:46.834 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:46.835 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:46.835 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:46.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:46.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:46.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:46.836 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:46.836 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:46.837 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:46.837 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:46.837 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:46.837 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:46.838 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:46.838 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:46.838 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:46.839 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:46.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:46.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:46.840 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:46.840 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:46.840 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:46.840 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:46.841 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:46.841 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:46.841 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:46.841 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:46.842 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:46.842 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:46.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:46.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:46.843 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:46.843 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:46.843 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:46.843 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:46.844 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:46.844 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:46.844 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:46.844 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:46.845 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:46.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:46.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:46.845 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:46.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:46.846 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:46.847 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:46.847 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:46.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:46.847 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:46Z" 2022-07-18T03:38:46.848 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:46.849 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:46Z" } 2022-07-18T03:38:46.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:46.851 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:46.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:46.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:51.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:51.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:51.854 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:51.904 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:51.905 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:51.968 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:51.969 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:51.969 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:51.969 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:51.969 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:51.970 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:51.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:51.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:51.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:51.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:51.981 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:51.981 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:51.982 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:51.982 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:51.982 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:51.982 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:51.983 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:51.983 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:51.983 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:51.983 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:51.984 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:51.984 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:51.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:51.984 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:51.985 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:51.985 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:51.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:51.985 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:51Z" 2022-07-18T03:38:51.986 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:51.987 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:51Z" } 2022-07-18T03:38:51.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:51.989 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:51.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:51.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:38:56.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:56.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:38:56.993 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:38:57.043 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:38:57.043 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:38:57.117 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:38:57.117 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:38:57.118 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:38:57.119 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:38:57.119 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:38:57.120 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:38:57.120 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:57.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:57.121 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:38:57.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:57.122 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:38:57.123 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:57.123 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:57.124 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:38:57.124 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:57.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:38:57.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:57.126 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:38:57.126 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:57.126 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:57.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:57.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:57.127 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:38:57.127 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:57.128 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:57.128 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:38:57.128 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:57.128 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:57.129 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:57.129 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:38:57.129 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:57.130 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:57.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:57.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:57.132 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:38:57.132 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:57.133 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:57.133 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:38:57.134 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:57.134 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:57.134 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:57.135 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:38:57.135 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:57.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:57.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:57.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:38:57.137 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:38:57.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:38:57.137 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:38:57.137 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:38:57.138 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:38:57.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:38:57.138 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:38:57.138 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:38:57.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:38:57.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:57.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:38:57.140 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:38:57.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:57.140 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:38:57.140 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:38:57.141 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:38:57.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:38:57.141 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:38:57Z" 2022-07-18T03:38:57.141 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:38:57.143 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:38:57Z" } 2022-07-18T03:38:57.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:38:57.144 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:38:57.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:38:57.145 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:02.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:02.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:02.148 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:02.203 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:02.204 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:02.266 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:02.267 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:02.267 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:02.268 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:02.268 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:02.268 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:02.268 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:02.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:02.269 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:02.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:02.270 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:02.270 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:02.271 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:02.271 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:02.271 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:02.271 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:02.272 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:02.272 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:02.272 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:02.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:02.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:02.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:02.273 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:02Z" 2022-07-18T03:39:02.285 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:02.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:02Z" } 2022-07-18T03:39:02.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:02.287 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:02.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:02.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:07.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:07.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:07.292 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:07.340 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:07.341 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:07.412 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:07.413 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:07.414 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:07.414 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:07.415 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:07.415 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:07.416 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:07.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:07.417 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:07.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:07.419 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:07.419 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:07.419 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:07.420 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:07.420 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:07.421 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:07.421 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:07.421 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:07.421 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:07.422 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:07.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:07.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:07.423 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:07.424 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:07.424 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:07.424 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:07.424 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:07.425 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:07.425 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:07.425 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:07.425 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:07.426 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:07.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:07.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:07.426 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:07.427 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:07.427 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:07.427 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:07.428 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:07.428 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:07.428 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:07.429 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:07.429 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:07.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:07.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:07.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:07.430 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:07.431 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:07.431 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:07.431 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:07.431 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:07.432 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:07.432 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:07.432 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:07.432 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:07.433 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:07.433 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:07.433 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:07.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:07.434 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:07.434 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:07.434 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:07.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:07.435 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:07Z" 2022-07-18T03:39:07.435 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:07.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:07Z" } 2022-07-18T03:39:07.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:07.438 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:07.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:07.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:12.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:12.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:12.444 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:12.492 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:12.493 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:12.551 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:12.551 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:12.551 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:12.552 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:12.552 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:12.552 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:12.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:12.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:12.553 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:12.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:12.554 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:12.554 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:12.554 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:12.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:12.555 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:12.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:12.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:12.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:12.556 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:12.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:12.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:12.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:12.557 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:12.557 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:12.558 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:12.558 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:12.558 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:12.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:12.559 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:12.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:12.559 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:12.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:12.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:12.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:12.560 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:12.561 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:12.561 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:12.561 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:12.561 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:12.562 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:12.562 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:12.563 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:12.563 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:12.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:12.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:12.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:12.564 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:12.565 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:12.565 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:12.566 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:12.566 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:12.566 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:12.566 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:12.567 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:12.567 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:12.567 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:12.567 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:12.568 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:12.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:12.568 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:12.568 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:12.569 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:12.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:12.569 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:12Z" 2022-07-18T03:39:12.570 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:12.571 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:12Z" } 2022-07-18T03:39:12.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:12.573 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:12.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:12.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:17.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:17.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:17.578 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:17.627 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:17.628 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:17.693 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:17Z" 2022-07-18T03:39:17.711 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:17.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:17Z" } 2022-07-18T03:39:17.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:17.714 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:17.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:17.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:22.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:22.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:22.720 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:22.776 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:22.777 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:22.852 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:22.853 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:22.853 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:22.853 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:22.854 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:22.854 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:22.854 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:22.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:22.855 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:22.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:22.856 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:22.856 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:22.856 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:22.857 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:22.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:22.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:22.857 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:22.858 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:22.858 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:22.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:22.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:22.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:22.859 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:22.859 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:22.860 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:22.860 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:22.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:22.861 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:22.861 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:22.862 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:22.862 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:22.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:22.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:22.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:22.863 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:22.863 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:22.863 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:22.864 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:22.864 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:22.864 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:22.865 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:22.865 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:22.865 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:22.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:22.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:22.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:22.867 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:22.867 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:22.867 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:22.867 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:22.868 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:22.868 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:22.868 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:22.868 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:22.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:22.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:22.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:22.869 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:22.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:22.870 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:22.871 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:22.871 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:22.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:22.871 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:22Z" 2022-07-18T03:39:22.872 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:22.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:22Z" } 2022-07-18T03:39:22.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:22.875 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:22.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:22.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:27.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:27.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:27.879 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:27.933 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:27.934 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:28.011 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:28.011 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:28.012 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:28.012 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:28.012 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:28.012 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:28.013 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:28.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:28.013 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:28.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:28.014 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:28.014 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:28.014 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:28.015 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:28.015 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:28.015 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:28.015 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:28.016 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:28.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:28.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:28.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:28.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:28.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:28.017 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:28.017 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:28.018 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:28.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:28.018 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:28.018 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:28.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:28.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:28.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:28.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:28.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:28.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:28.020 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:28.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:28.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:28.021 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:28.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:28.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:28.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:28.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:28.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:28.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:28.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:28.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:28.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:28.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:28.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:28.024 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:28.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:28.025 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:28.025 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:28.026 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:28.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:28.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:28.027 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:28.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:28.028 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:28.029 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:28.029 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:28.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:28.030 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:27Z" 2022-07-18T03:39:28.031 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:28.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:27Z" } 2022-07-18T03:39:28.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:28.034 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:28.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:28.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:33.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:33.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:33.039 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:33.086 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:33.087 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:33.144 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:33.144 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:33.144 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:33.145 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:33.145 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:33.145 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:33.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:33.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:33.146 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:33.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:33.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:33.147 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:33.148 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:33.148 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:33.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:33.148 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:33.149 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:33.149 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:33.150 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:33.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:33.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:33.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:33.151 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:33.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:33.151 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:33.152 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:33.152 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:33.152 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:33.152 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:33.153 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:33.153 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:33.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:33.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:33.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:33.154 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:33.154 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:33.155 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:33.155 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:33.155 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:33.155 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:33.156 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:33.156 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:33.156 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:33.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:33.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:33.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:33.157 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:33.157 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:33.158 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:33.158 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:33.159 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:33.159 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:33.160 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:33.160 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:33.161 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:33.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:33.161 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:33.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:33.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:33.163 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:33.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:33.163 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:33.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:33.164 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:33Z" 2022-07-18T03:39:33.165 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:33.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:33Z" } 2022-07-18T03:39:33.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:33.169 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:33.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:33.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:38.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:38.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:38.173 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:38.226 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:38.226 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:38.295 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:38.296 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:38.296 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:38.297 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:38.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:38.299 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:38.300 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:38.301 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:38.301 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:38.301 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:38.301 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:38.302 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:38.304 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:38.305 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:38.306 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:38.306 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:38.306 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:38.307 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:38.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:38.307 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:38.307 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:38.308 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:38.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:38.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:38.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:38.309 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:38.309 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:38.309 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:38.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:38.310 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:38.311 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:38.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:38.312 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:38.312 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:38.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:38.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:38.314 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:38.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:38.314 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:38.315 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:38.315 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:38.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:38.316 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:38Z" 2022-07-18T03:39:38.317 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:38.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:38Z" } 2022-07-18T03:39:38.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:38.320 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:38.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:38.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:43.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:43.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:43.338 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:43.409 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:43.409 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:43.544 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:43.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:43.559 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:43.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:43.559 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:43.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:43.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:43.560 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:43.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:43.561 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:43.561 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:43Z" 2022-07-18T03:39:43.562 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:43.564 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:43Z" } 2022-07-18T03:39:43.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:43.565 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:43.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:43.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:48.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:48.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:48.569 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:48.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:48.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:48.714 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:48Z" 2022-07-18T03:39:48.730 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:48.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:48Z" } 2022-07-18T03:39:48.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:48.733 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:48.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:48.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:53.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:53.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:53.737 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:53.793 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:53.794 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:53.882 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:53.883 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:53.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:53.885 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:53.886 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:53.887 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:53.887 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:53.887 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:53.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:53.889 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:53.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:53.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:53.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:53.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:53.892 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:53.893 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:53.893 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:53.894 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:53.894 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:53.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:53.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:53.896 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:53.897 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:53.897 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:53.897 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:53.897 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:53.898 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:53.898 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:53.898 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:53.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:53.899 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:53.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:53.899 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:53.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:53.900 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:53.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:53.900 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:53Z" 2022-07-18T03:39:53.901 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:53.902 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:53Z" } 2022-07-18T03:39:53.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:53.903 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:53.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:53.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:39:58.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:58.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:39:58.907 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:39:58.963 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:39:58.964 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:39:59.029 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:39:59.029 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:39:59.029 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:39:59.030 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:39:59.030 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:39:59.030 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:39:59.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:59.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:59.031 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:39:59.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:59.032 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:39:59.032 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:59.032 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:59.033 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:39:59.033 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:59.033 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:39:59.033 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:59.034 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:39:59.034 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:59.034 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:59.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:59.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:59.035 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:39:59.035 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:59.035 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:59.036 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:39:59.036 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:59.036 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:59.036 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:59.037 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:39:59.037 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:59.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:59.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:59.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:59.038 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:39:59.039 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:59.039 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:59.039 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:39:59.039 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:59.040 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:59.040 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:59.040 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:39:59.041 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:59.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:59.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:59.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:39:59.042 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:39:59.042 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:39:59.042 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:39:59.043 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:39:59.043 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:39:59.043 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:39:59.043 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:39:59.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:39:59.044 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:39:59.044 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:59.044 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:39:59.045 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:39:59.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:59.045 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:39:59.045 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:39:59.045 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:39:59.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:39:59.046 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:39:59Z" 2022-07-18T03:39:59.046 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:39:59.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:39:59Z" } 2022-07-18T03:39:59.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:39:59.049 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:39:59.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:39:59.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:04.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:04.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:04.054 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:04.111 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:04.111 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:04.190 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:04.191 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:04.192 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:04.192 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:04.193 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:04.193 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:04.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:04.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:04.195 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:04.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:04.196 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:04.196 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:04.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:04.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:04.198 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:04.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:04.199 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:04.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:04.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:04.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:04.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:04.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:04.202 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:04.203 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:04.203 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:04.204 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:04.204 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:04.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:04.205 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:04.205 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:04.206 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:04.206 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:04.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:04.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:04.208 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:04.208 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:04.208 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:04.209 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:04.209 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:04.210 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:04.210 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:04.211 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:04.211 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:04.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:04.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:04.212 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:04.213 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:04.213 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:04.213 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:04.214 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:04.214 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:04.214 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:04.215 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:04.215 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:04.215 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:04.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:04.216 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:04.216 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:04.216 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:04.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:04.217 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:04Z" 2022-07-18T03:40:04.217 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:04.219 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:04Z" } 2022-07-18T03:40:04.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:04.220 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:04.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:04.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:09.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:09.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:09.225 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:09.282 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:09.282 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:09.347 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:09Z" 2022-07-18T03:40:09.366 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:09.368 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:09Z" } 2022-07-18T03:40:09.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:09.369 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:09.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:09.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:14.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:14.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:14.374 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:14.431 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:14.432 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:14.496 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:14.496 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:14.497 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:14.497 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:14.498 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:14.498 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:14.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:14.499 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:14.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:14.499 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:14.500 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:14.500 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:14.500 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:14.500 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:14.501 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:14.501 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:14.501 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:14.501 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:14.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:14.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:14.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:14.503 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:14.503 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:14.503 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:14.504 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:14.504 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:14.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:14.505 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:14.505 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:14.505 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:14.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:14.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:14.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:14.506 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:14.507 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:14.507 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:14.507 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:14.507 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:14.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:14.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:14.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:14.509 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:14.510 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:14.510 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:14.510 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:14.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:14.511 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:14.511 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:14.511 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:14.512 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:14.513 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:14.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:14.514 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:14Z" 2022-07-18T03:40:14.514 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:14.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:14Z" } 2022-07-18T03:40:14.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:14.517 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:14.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:14.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:19.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:19.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:19.521 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:19.575 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:19.576 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:19.656 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:19.657 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:19.658 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:19.658 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:19.659 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:19.659 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:19.660 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:19.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:19.661 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:19.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:19.662 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:19.662 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:19.663 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:19.663 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:19.664 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:19.664 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:19.665 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:19.665 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:19.666 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:19.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:19.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:19.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:19.668 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:19.668 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:19.669 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:19.669 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:19.670 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:19.670 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:19.671 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:19.671 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:19.672 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:19.672 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:19.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:19.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:19.673 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:19.674 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:19.674 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:19.675 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:19.675 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:19.676 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:19.676 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:19.676 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:19.677 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:19.677 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:19.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:19.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:19.679 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:19.679 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:19.680 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:19.681 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:19.682 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:19.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:19.683 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:19.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:19.684 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:19.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:19.685 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:19.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:19.686 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:19Z" 2022-07-18T03:40:19.686 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:19.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:19Z" } 2022-07-18T03:40:19.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:19.689 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:19.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:19.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:24.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:24.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:24.693 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:24.748 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:24.749 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:24.818 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:24Z" 2022-07-18T03:40:24.839 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:24.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:24Z" } 2022-07-18T03:40:24.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:24.841 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:24.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:24.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:29.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:29.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:29.846 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:29.898 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:29.899 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:29.976 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:29.976 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:29.976 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:29.977 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:29.977 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:29.977 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:29.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:29.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:29.978 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:29.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:29.979 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:29.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:29.979 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:29.979 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:29.980 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:29.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:29.980 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:29.980 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:29.981 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:29.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:29.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:29.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:29.982 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:29.983 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:29.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:29.985 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:29.986 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:29.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:29.988 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:29.989 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:29.990 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:29.990 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:29.990 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:29.990 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:29.991 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:29.992 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:29.992 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:29.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:29.992 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:29Z" 2022-07-18T03:40:29.993 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:29.994 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:29Z" } 2022-07-18T03:40:29.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:29.995 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:29.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:29.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:34.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:34.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:34.999 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:35.056 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:35.057 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:35.148 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:35.149 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:35.149 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:35.149 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:35.149 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:35.150 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:35.150 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:35.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:35.150 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:35.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:35.151 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:35.151 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:35.152 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:35.152 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:35.152 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:35.152 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:35.153 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:35.153 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:35.153 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:35.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:35.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:35.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:35.154 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:35.154 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:35.155 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:35.155 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:35.155 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:35.155 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:35.156 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:35.156 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:35.156 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:35.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:35.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:35.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:35.157 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:35.157 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:35.157 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:35.158 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:35.158 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:35.158 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:35.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:35.159 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:35.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:35.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:35.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:35.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:35.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:35.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:35.160 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:35.161 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:35.161 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:35.161 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:35.161 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:35.162 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:35.162 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:35.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:35.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:35.163 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:35.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:35.163 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:35.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:35.164 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:35.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:35.164 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:35Z" 2022-07-18T03:40:35.164 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:35.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:35Z" } 2022-07-18T03:40:35.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:35.167 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:35.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:35.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:40.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:40.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:40.171 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:40.227 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:40.227 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:40.296 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:40.296 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:40.297 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:40.298 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:40.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:40.299 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:40.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:40.299 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:40.300 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:40.300 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:40.300 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:40.300 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:40.301 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:40.301 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:40.301 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:40.301 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:40.302 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:40.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:40.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:40.302 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:40.303 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:40.303 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:40.303 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:40.313 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:40.313 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:40.314 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:40.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:40.315 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:40Z" 2022-07-18T03:40:40.315 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:40.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:40Z" } 2022-07-18T03:40:40.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:40.317 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:40.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:40.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:45.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:45.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:45.323 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:45.378 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:45.379 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:45.450 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:45.450 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:45.451 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:45.451 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:45.451 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:45.451 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:45.452 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:45.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:45.452 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:45.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:45.453 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:45.453 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:45.456 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:45.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:45.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:45.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:45.457 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:45.458 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:45.459 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:45.459 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:45.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:45.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:45.464 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:45.464 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:45Z" 2022-07-18T03:40:45.472 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:45.474 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:45Z" } 2022-07-18T03:40:45.474 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:45.475 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:45.475 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:45.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:50.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:50.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:50.480 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:50.535 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:50.536 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:50.610 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:50.613 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:50.613 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:50.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:50.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:50.627 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:50.628 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:50.628 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:50.629 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:50.629 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:50.630 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:50.630 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:50.631 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:50.631 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:50.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:50.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:50.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:50.633 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:50.633 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:50.634 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:50.635 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:50.635 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:50.635 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:50.644 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:50.645 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:50.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:50.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:50.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:50.647 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:50.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:50.647 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:50.648 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:50.648 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:50.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:50.649 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:50Z" 2022-07-18T03:40:50.649 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:50.651 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:50Z" } 2022-07-18T03:40:50.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:50.653 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:50.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:50.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:40:55.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:55.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:40:55.659 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:40:55.710 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:40:55.711 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:40:55.786 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:40:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:40:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:40:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:40:55.787 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:40:55.788 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:40:55.788 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:55.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:55.788 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:40:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:40:55.789 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:40:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:55.790 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:40:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:40:55.791 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:55.792 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:55.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:55.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:55.792 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:40:55.793 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:55.793 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:55.793 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:40:55.793 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:55.794 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:55.794 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:55.794 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:40:55.794 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:55.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:55.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:55.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:55.796 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:40:55.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:55.796 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:55.796 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:40:55.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:55.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:55.797 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:55.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:40:55.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:55.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:55.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:55.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:40:55.799 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:40:55.799 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:40:55.799 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:40:55.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:40:55.800 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:40:55.800 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:40:55.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:40:55.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:40:55.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:40:55.801 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:55.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:40:55.802 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:40:55.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:55.802 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:40:55.803 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:40:55.803 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:40:55.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:40:55.804 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:40:55Z" 2022-07-18T03:40:55.804 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:40:55.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:40:55Z" } 2022-07-18T03:40:55.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:40:55.807 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:40:55.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:40:55.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:00.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:00.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:00.811 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:00.859 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:00.860 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:00.933 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:00.933 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:00.934 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:00.934 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:00.934 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:00.934 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:00.935 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:00.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:00.935 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:00.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:00.937 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:00.937 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:00.937 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:00.937 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:00.938 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:00.938 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:00.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:00.939 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:00.940 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:00.940 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:00.940 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:00.940 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:00.941 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:00.941 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:00.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:00.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:00.942 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:00.942 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:00.943 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:00.944 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:00.944 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:00.944 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:00.944 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:00.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:00.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:00.946 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:00.946 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:00.946 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:00.946 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:00.947 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:00.947 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:00.947 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:00.947 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:00.948 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:00.948 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:00.948 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:00.949 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:00.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:00.949 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:00.949 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:00.950 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:00.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:00.950 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:00Z" 2022-07-18T03:41:00.950 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:00.952 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:00Z" } 2022-07-18T03:41:00.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:00.953 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:00.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:00.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:05.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:05.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:05.958 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:06.016 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:06.017 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:06.080 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:06.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:06.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:06.086 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:06.086 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:06.087 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:06.087 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:06.087 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:06.087 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:06.088 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:06.088 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:06.088 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:06.088 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:06.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:06.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:06.089 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:06.089 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:06.090 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:06.090 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:06.090 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:06.090 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:06.091 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:06.091 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:06.091 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:06.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:06.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:06.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:06.092 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:06.092 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:06.093 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:06.093 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:06.093 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:06.094 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:06.094 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:06.094 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:06.094 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:06.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:06.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:06.095 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:06.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:06.096 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:06.096 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:06.096 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:06.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:06.097 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:06Z" 2022-07-18T03:41:06.097 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:06.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:06Z" } 2022-07-18T03:41:06.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:06.099 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:06.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:06.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:11.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:11.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:11.103 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:11.158 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:11.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:11.252 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:11.252 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:11.252 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:11.252 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:11.253 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:11.253 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:11.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:11.254 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:11.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:11.254 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:11.254 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:11.255 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:11.255 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:11.255 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:11.255 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:11.256 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:11.256 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:11.257 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:11.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:11.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:11.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:11.259 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:11.259 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:11.260 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:11.260 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:11.260 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:11.260 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:11.261 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:11.261 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:11.262 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:11.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:11.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:11.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:11.263 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:11.263 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:11.263 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:11.263 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:11.265 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:11.265 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:11.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:11.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:11.266 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:11.266 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:11.266 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:11.266 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:11.267 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:11.267 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:11.267 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:11.267 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:11.268 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:11.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:11.270 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:11.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:11.271 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:11Z" 2022-07-18T03:41:11.271 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:11.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:11Z" } 2022-07-18T03:41:11.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:11.276 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:11.276 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:11.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:16.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:16.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:16.281 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:16.330 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:16.331 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:16.401 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:16.401 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:16.402 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:16.402 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:16.402 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:16.402 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:16.403 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:16.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:16.403 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:16.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:16.404 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:16.404 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:16.404 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:16.404 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:16.405 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:16.405 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:16.405 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:16.405 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:16.406 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:16.406 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:16.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:16.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:16.407 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:16.407 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:16.407 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:16.408 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:16.408 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:16.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:16.408 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:16.408 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:16.409 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:16.409 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:16.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:16.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:16.410 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:16.410 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:16.410 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:16.410 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:16.411 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:16.411 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:16.411 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:16.411 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:16.412 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:16.412 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:16.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:16.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:16.413 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:16.413 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:16.413 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:16.413 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:16.414 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:16.414 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:16.414 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:16.415 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:16.415 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:16.415 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:16.416 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:16.416 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:16.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:16.416 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:16.417 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:16.417 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:16.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:16.417 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:16Z" 2022-07-18T03:41:16.418 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:16.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:16Z" } 2022-07-18T03:41:16.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:16.420 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:16.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:16.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:21.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:21.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:21.424 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:21.479 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:21.480 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:21.545 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:21.546 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:21.546 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:21.546 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:21.546 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:21.547 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:21.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:21.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:21.548 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:21.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:21.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:21.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:21.549 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:21.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:21.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:21.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:21.550 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:21.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:21.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:21.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:21.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:21.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:21.551 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:21.551 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:21.552 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:21.552 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:21.552 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:21.552 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:21.553 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:21.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:21.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:21.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:21.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:21.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:21.554 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:21.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:21.555 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:21.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:21.555 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:21.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:21.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:21.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:21.556 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:21.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:21.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:21.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:21.558 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:21.558 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:21.559 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:21.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:21.560 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:21.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:21.560 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:21.561 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:21.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:21.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:21.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:21.562 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:21.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:21.562 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:21.562 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:21.563 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:21.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:21.563 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:21Z" 2022-07-18T03:41:21.563 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:21.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:21Z" } 2022-07-18T03:41:21.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:21.566 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:21.566 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:21.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:26.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:26.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:26.570 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:26.618 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:26.619 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:26.673 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:26.673 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:26.673 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:26.677 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:26.677 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:26.677 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:26.678 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:26.678 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:26.678 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:26.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:26.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:26.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:26.680 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:26.681 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:26.682 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:26.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:26.684 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:26.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:26.686 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:26.687 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:26.688 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:26.689 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:26.690 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:26.691 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:26Z" 2022-07-18T03:41:26.691 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:26.692 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:26Z" } 2022-07-18T03:41:26.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:26.694 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:26.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:26.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:31.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:31.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:31.698 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:31.747 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:31.748 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:31.808 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:31.812 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:31.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:31.812 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:31.812 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:31.813 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:31.814 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:31.814 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:31.814 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:31.814 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:31.815 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:31.815 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:31.815 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:31.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:31.816 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:31.817 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:31.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:31.819 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:31.820 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:31.821 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:31.821 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:31.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:31.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:31.822 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:31.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:31.822 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:31.822 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:31.823 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:31.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:31.823 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:31Z" 2022-07-18T03:41:31.823 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:31.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:31Z" } 2022-07-18T03:41:31.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:31.826 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:31.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:31.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:36.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:36.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:36.829 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:36.884 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:36.885 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:36.975 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:36.976 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:36.976 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:36.976 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:36.977 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:36.977 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:36.977 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:36.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:36.978 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:36.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:36.978 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:36.978 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:36.979 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:36.979 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:36.979 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:36.979 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:36.980 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:36.980 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:36.980 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:36.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:36.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:36.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:36.981 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:36.982 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:36.982 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:36.982 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:36.982 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:36.983 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:36.983 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:36.983 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:36.984 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:36.984 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:36.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:36.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:36.985 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:36.985 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:36.985 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:36.985 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:36.986 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:36.986 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:36.986 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:36.987 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:36.987 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:36.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:36.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:36.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:36.988 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:36.988 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:36.989 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:36.989 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:36.990 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:36.990 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:36.990 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:36.991 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:36.991 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:36.991 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:36.991 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:36.991 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:36.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:36.992 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:36.992 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:36.993 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:36.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:36.993 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:36Z" 2022-07-18T03:41:36.993 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:36.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:36Z" } 2022-07-18T03:41:36.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:36.996 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:36.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:36.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:41.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:41.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:42.000 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:42.055 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:42.055 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:42.122 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:42.122 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:42.122 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:42.123 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:42.123 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:42.123 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:42.124 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:42.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:42.124 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:42.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:42.125 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:42.125 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:42.125 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:42.126 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:42.126 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:42.126 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:42.127 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:42.127 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:42.127 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:42.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:42.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:42.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:42.128 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:42.128 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:42.128 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:42.129 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:42.129 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:42.129 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:42.129 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:42.130 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:42.130 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:42.130 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:42.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:42.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:42.131 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:42.131 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:42.132 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:42.132 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:42.132 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:42.132 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:42.133 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:42.133 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:42.133 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:42.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:42.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:42.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:42.134 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:42.134 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:42.135 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:42.135 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:42.135 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:42.135 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:42.136 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:42.136 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:42.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:42.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:42.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:42.137 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:42.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:42.137 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:42.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:42.138 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:42.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:42.138 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:42Z" 2022-07-18T03:41:42.139 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:42.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:42Z" } 2022-07-18T03:41:42.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:42.141 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:42.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:42.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:47.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:47.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:47.145 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:47.201 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:47.201 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:47.291 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:47.292 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:47.292 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:47.292 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:47.292 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:47.293 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:47.293 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:47.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:47.293 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:47.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:47.294 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:47.294 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:47.295 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:47.295 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:47.295 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:47.295 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:47.296 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:47.296 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:47.296 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:47.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:47.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:47.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:47.297 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:47.297 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:47.298 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:47.298 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:47.298 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:47.301 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:47.301 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:47.301 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:47.301 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:47.302 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:47.302 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:47.302 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:47.303 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:47.303 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:47.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:47.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:47.304 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:47.304 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:47.304 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:47.304 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:47.305 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:47.305 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:47.305 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:47.305 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:47.306 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:47.306 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:47.306 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:47.306 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:47.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:47.307 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:47.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:47.307 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:47.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:47.308 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:47Z" 2022-07-18T03:41:47.308 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:47.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:47Z" } 2022-07-18T03:41:47.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:47.311 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:47.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:47.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:52.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:52.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:52.315 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:52.370 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:52.371 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:52.437 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:52.437 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:52.438 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:52.438 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:52.438 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:52.439 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:52.439 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:52.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:52.439 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:52.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:52.440 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:52.440 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:52.440 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:52.441 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:52.441 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:52.441 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:52.442 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:52.442 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:52.442 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:52.442 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:52.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:52.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:52.443 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:52.443 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:52.444 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:52.444 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:52.444 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:52.444 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:52.445 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:52.445 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:52.445 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:52.449 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:52.449 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:52.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:52.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:52.450 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:52.451 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:52.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:52.453 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:52.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:52.454 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:52Z" 2022-07-18T03:41:52.454 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:52.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:52Z" } 2022-07-18T03:41:52.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:52.457 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:52.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:52.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:41:57.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:57.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:41:57.461 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:41:57.510 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:41:57.511 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:41:57.583 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:41:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:41:57.583 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:41:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:41:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:41:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:41:57.584 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:41:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:57.585 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:41:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:57.586 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:41:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:41:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:57.587 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:41:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:57.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:41:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:57.589 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:41:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:57.590 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:41:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:57.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:41:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:57.592 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:41:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:57.593 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:57.594 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:41:57.594 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:57.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:41:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:41:57.595 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:41:57.596 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:41:57.596 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:41:57.596 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:41:57.596 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:41:57.597 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:41:57.597 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:41:57.597 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:41:57.597 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:57.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:41:57.598 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:41:57.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:57.598 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:41:57.599 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:41:57.599 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:41:57.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:41:57.599 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:41:57Z" 2022-07-18T03:41:57.600 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:41:57.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:41:57Z" } 2022-07-18T03:41:57.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:41:57.603 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:41:57.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:41:57.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:02.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:02.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:02.606 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:02.661 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:02.662 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:02.729 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:02.729 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:02.732 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:02.733 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:02.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:02.735 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:02.736 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:02.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:02.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:02.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:02.738 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:02.738 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:02.738 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:02.739 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:02.739 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:02.739 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:02.739 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:02.740 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:02.741 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:02.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:02.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:02.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:02.742 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:02.742 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:02.742 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:02.743 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:02.743 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:02.743 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:02.743 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:02.744 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:02.744 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:02.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:02.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:02.745 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:02.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:02.745 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:02.746 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:02.746 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:02.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:02.746 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:02Z" 2022-07-18T03:42:02.747 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:02.748 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:02Z" } 2022-07-18T03:42:02.749 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:02.749 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:02.749 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:02.750 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:07.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:07.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:07.752 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:07.808 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:07.809 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:07.884 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:07.892 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:07.893 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:07.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:07.895 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:07.896 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:07.897 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:07.898 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:07.899 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:07.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:07.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:07.902 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:07.903 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:07.903 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:07.904 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:07.905 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:07.905 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:07.906 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:07.906 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:07.907 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:07.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:07.908 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:07.909 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:07.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:07.910 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:07.910 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:07.911 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:07.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:07.912 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:07Z" 2022-07-18T03:42:07.912 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:07.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:07Z" } 2022-07-18T03:42:07.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:07.917 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:07.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:07.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:12.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:12.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:12.920 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:12.978 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:12.979 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:13.048 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:13.049 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:13.049 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:13.049 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:13.049 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:13.050 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:13.050 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:13.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:13.050 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:13.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:13.051 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:13.051 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:13.051 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:13.052 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:13.052 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:13.052 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:13.052 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:13.053 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:13.053 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:13.053 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:13.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:13.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:13.054 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:13.054 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:13.065 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:13Z" 2022-07-18T03:42:13.065 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:13.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:13Z" } 2022-07-18T03:42:13.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:13.067 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:13.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:13.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:18.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:18.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:18.071 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:18.126 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:18.127 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:18.201 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:18.201 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:18.202 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:18.203 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:18.204 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:18.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:18.205 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:18.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:18.206 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:18.206 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:18.207 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:18.207 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:18.208 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:18.208 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:18.209 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:18.209 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:18.209 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:18.210 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:18.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:18.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:18.211 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:18.212 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:18.212 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:18.212 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:18.213 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:18.213 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:18.213 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:18.213 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:18.214 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:18.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:18.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:18.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:18.215 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:18.215 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:18.215 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:18.216 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:18.216 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:18.216 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:18.216 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:18.217 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:18.217 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:18.217 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:18.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:18.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:18.218 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:18.218 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:18.218 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:18.219 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:18.219 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:18.219 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:18.219 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:18.220 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:18.220 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:18.220 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:18.221 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:18.221 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:18.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:18.222 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:18.222 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:18.222 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:18.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:18.223 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:18Z" 2022-07-18T03:42:18.223 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:18.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:18Z" } 2022-07-18T03:42:18.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:18.226 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:18.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:18.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:23.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:23.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:23.230 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:23.280 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:23.280 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:23.367 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:23Z" 2022-07-18T03:42:23.387 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:23.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:23Z" } 2022-07-18T03:42:23.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:23.390 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:23.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:23.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:28.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:28.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:28.394 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:28.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:28.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:28.514 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:28.514 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:28.515 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:28.515 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:28.516 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:28.517 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:28.517 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:28.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:28.518 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:28.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:28.519 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:28.519 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:28.520 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:28.520 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:28.520 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:28.520 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:28Z" 2022-07-18T03:42:28.534 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:28.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:28Z" } 2022-07-18T03:42:28.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:28.536 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:28.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:28.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:33.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:33.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:33.540 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:33.597 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:33.598 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:33.680 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:33.681 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:33.681 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:33.681 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:33.682 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:33.682 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:33.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:33.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:33.683 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:33.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:33.683 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:33.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:33.684 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:33.684 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:33.684 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:33.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:33.690 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:33.690 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:33.690 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:33.695 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:33.695 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:33.695 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:33.695 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:33.696 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:33.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:33.696 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:33.696 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:33Z" 2022-07-18T03:42:33.697 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:33.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:33Z" } 2022-07-18T03:42:33.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:33.700 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:33.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:33.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:38.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:38.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:38.704 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:38.759 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:38.760 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:38.822 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:38.822 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:38.823 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:38.824 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:38.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:38.824 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:38.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:38.825 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:38.825 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:38.825 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:38.826 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:38.826 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:38.826 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:38.826 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:38.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:38.828 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:38.829 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:38Z" 2022-07-18T03:42:38.838 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:38.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:38Z" } 2022-07-18T03:42:38.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:38.841 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:38.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:38.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:43.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:43.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:43.846 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:43.895 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:43.895 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:43.966 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:43.967 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:43.967 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:43.967 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:43.968 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:43.968 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:43.968 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:43.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:43.969 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:43.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:43.969 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:43.969 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:43.970 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:43.970 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:43.970 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:43.971 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:43.971 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:43.971 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:43.971 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:43.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:43.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:43.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:43.972 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:43.973 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:43.973 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:43.973 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:43.973 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:43.974 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:43.974 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:43.974 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:43.975 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:43.975 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:43.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:43.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:43.976 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:43.976 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:43.976 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:43.978 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:43.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:43.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:43.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:43.979 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:43.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:43.980 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:43.980 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:43.980 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:43.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:43.981 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:43.981 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:43.981 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:43.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:43.982 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:43.982 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:43.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:43.982 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:43.983 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:43.983 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:43.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:43.983 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:43Z" 2022-07-18T03:42:43.984 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:43.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:43Z" } 2022-07-18T03:42:43.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:43.986 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:43.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:43.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:48.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:48.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:48.990 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:49.045 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:49.046 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:49.133 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:49Z" 2022-07-18T03:42:49.150 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:49.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:49Z" } 2022-07-18T03:42:49.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:49.153 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:49.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:49.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:54.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:54.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:54.157 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:54.213 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:54.213 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:54.275 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:54.275 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115469.839831_166.1", 2022-07-18T03:42:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:37:49.839831Z", 2022-07-18T03:42:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:54.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:54.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:54.290 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:54.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:54.291 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:54.291 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:54.291 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:54.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:54.292 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:54Z" 2022-07-18T03:42:54.292 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:54.293 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115469.839831_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:37:49.839831Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:54Z" } 2022-07-18T03:42:54.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115469.839831_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:37:49.839831Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:54.294 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:54.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115469.839831_166.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:54.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:42:59.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... FAIL 2022-07-18T03:42:59.305 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-24 2022-07-18T03:42:59.306 DEBUG:boto:path=/mlvank-24/ 2022-07-18T03:42:59.306 DEBUG:boto:auth_path=/mlvank-24/ 2022-07-18T03:42:59.307 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.308 DEBUG:boto:Path: /mlvank-24/ 2022-07-18T03:42:59.310 DEBUG:boto:Data: 2022-07-18T03:42:59.311 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.311 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:42:59.312 DEBUG:boto:Port: 8000 2022-07-18T03:42:59.312 DEBUG:boto:Params: {} 2022-07-18T03:42:59.313 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:42:59.314 DEBUG:boto:Token: None 2022-07-18T03:42:59.315 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-24/ 2022-07-18T03:42:59.316 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:b27TMAiHxTBlBQBFTfT3rMXI4CA= 2022-07-18T03:42:59.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:b27TMAiHxTBlBQBFTfT3rMXI4CA=', 'Content-Length': '0'} 2022-07-18T03:42:59.333 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1f9268290bd40ad-0062d4d6c3-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.334 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-25 2022-07-18T03:42:59.335 DEBUG:boto:path=/mlvank-25/ 2022-07-18T03:42:59.335 DEBUG:boto:auth_path=/mlvank-25/ 2022-07-18T03:42:59.336 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.336 DEBUG:boto:Path: /mlvank-25/ 2022-07-18T03:42:59.337 DEBUG:boto:Data: 2022-07-18T03:42:59.337 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.338 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:42:59.338 DEBUG:boto:Port: 8000 2022-07-18T03:42:59.339 DEBUG:boto:Params: {} 2022-07-18T03:42:59.340 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:42:59.341 DEBUG:boto:Token: None 2022-07-18T03:42:59.341 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-25/ 2022-07-18T03:42:59.342 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:bD9xmbcA8PWP4hAMWxSZLHCNG7I= 2022-07-18T03:42:59.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:bD9xmbcA8PWP4hAMWxSZLHCNG7I=', 'Content-Length': '0'} 2022-07-18T03:42:59.380 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003fe9399f51e80b75-0062d4d6c3-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.381 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-26 2022-07-18T03:42:59.381 DEBUG:boto:path=/mlvank-26/ 2022-07-18T03:42:59.382 DEBUG:boto:auth_path=/mlvank-26/ 2022-07-18T03:42:59.382 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.383 DEBUG:boto:Path: /mlvank-26/ 2022-07-18T03:42:59.383 DEBUG:boto:Data: 2022-07-18T03:42:59.384 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.384 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:42:59.384 DEBUG:boto:Port: 8001 2022-07-18T03:42:59.385 DEBUG:boto:Params: {} 2022-07-18T03:42:59.386 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:42:59.386 DEBUG:boto:Token: None 2022-07-18T03:42:59.387 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-26/ 2022-07-18T03:42:59.387 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:OVFoioyqeb0QOJrGs1pffqNyT28= 2022-07-18T03:42:59.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:OVFoioyqeb0QOJrGs1pffqNyT28=', 'Content-Length': '0'} 2022-07-18T03:42:59.447 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000706e25205cd81876-0062d4d6c3-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.448 DEBUG:boto:path=/mlvank-24/ 2022-07-18T03:42:59.449 DEBUG:boto:auth_path=/mlvank-24/ 2022-07-18T03:42:59.450 DEBUG:boto:path=/mlvank-24/?versioning 2022-07-18T03:42:59.450 DEBUG:boto:auth_path=/mlvank-24/?versioning 2022-07-18T03:42:59.451 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.452 DEBUG:boto:Path: /mlvank-24/?versioning 2022-07-18T03:42:59.452 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:42:59.453 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.454 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:42:59.455 DEBUG:boto:Port: 8000 2022-07-18T03:42:59.455 DEBUG:boto:Params: {} 2022-07-18T03:42:59.456 DEBUG:boto:Token: None 2022-07-18T03:42:59.457 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-24/?versioning 2022-07-18T03:42:59.458 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:yYNX1uc8lAfACl7WtOw7IEpqmQQ= 2022-07-18T03:42:59.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:yYNX1uc8lAfACl7WtOw7IEpqmQQ=', 'Content-Length': '228'} 2022-07-18T03:42:59.473 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009c1a4e3f15a78885-0062d4d6c3-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.474 DEBUG:boto:path=/mlvank-24/ 2022-07-18T03:42:59.474 DEBUG:boto:auth_path=/mlvank-24/ 2022-07-18T03:42:59.475 DEBUG:boto:path=/mlvank-24/?versioning 2022-07-18T03:42:59.476 DEBUG:boto:auth_path=/mlvank-24/?versioning 2022-07-18T03:42:59.476 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.477 DEBUG:boto:Path: /mlvank-24/?versioning 2022-07-18T03:42:59.478 DEBUG:boto:Data: Suspended Disabled 2022-07-18T03:42:59.479 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.479 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:42:59.480 DEBUG:boto:Port: 8000 2022-07-18T03:42:59.481 DEBUG:boto:Params: {} 2022-07-18T03:42:59.482 DEBUG:boto:Token: None 2022-07-18T03:42:59.482 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-24/?versioning 2022-07-18T03:42:59.483 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:yYNX1uc8lAfACl7WtOw7IEpqmQQ= 2022-07-18T03:42:59.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:yYNX1uc8lAfACl7WtOw7IEpqmQQ=', 'Content-Length': '230'} 2022-07-18T03:42:59.492 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079168060843441d7-0062d4d6c3-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.493 DEBUG:boto:path=/mlvank-25/ 2022-07-18T03:42:59.494 DEBUG:boto:auth_path=/mlvank-25/ 2022-07-18T03:42:59.494 DEBUG:boto:path=/mlvank-25/?versioning 2022-07-18T03:42:59.495 DEBUG:boto:auth_path=/mlvank-25/?versioning 2022-07-18T03:42:59.496 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.497 DEBUG:boto:Path: /mlvank-25/?versioning 2022-07-18T03:42:59.497 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:42:59.498 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.499 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:42:59.500 DEBUG:boto:Port: 8000 2022-07-18T03:42:59.500 DEBUG:boto:Params: {} 2022-07-18T03:42:59.501 DEBUG:boto:Token: None 2022-07-18T03:42:59.502 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-25/?versioning 2022-07-18T03:42:59.503 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1ClDh+gb44wz2s5xMg8yJXY8D/M= 2022-07-18T03:42:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1ClDh+gb44wz2s5xMg8yJXY8D/M=', 'Content-Length': '228'} 2022-07-18T03:42:59.515 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb7cf2b222535332-0062d4d6c3-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.516 DEBUG:boto:path=/mlvank-25/ 2022-07-18T03:42:59.517 DEBUG:boto:auth_path=/mlvank-25/ 2022-07-18T03:42:59.517 DEBUG:boto:path=/mlvank-25/?versioning 2022-07-18T03:42:59.518 DEBUG:boto:auth_path=/mlvank-25/?versioning 2022-07-18T03:42:59.518 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.519 DEBUG:boto:Path: /mlvank-25/?versioning 2022-07-18T03:42:59.519 DEBUG:boto:Data: Suspended Disabled 2022-07-18T03:42:59.520 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.520 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:42:59.521 DEBUG:boto:Port: 8000 2022-07-18T03:42:59.521 DEBUG:boto:Params: {} 2022-07-18T03:42:59.522 DEBUG:boto:Token: None 2022-07-18T03:42:59.522 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-25/?versioning 2022-07-18T03:42:59.523 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1ClDh+gb44wz2s5xMg8yJXY8D/M= 2022-07-18T03:42:59.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1ClDh+gb44wz2s5xMg8yJXY8D/M=', 'Content-Length': '230'} 2022-07-18T03:42:59.532 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a5f6fa1b5b7f098-0062d4d6c3-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.533 DEBUG:boto:path=/mlvank-26/ 2022-07-18T03:42:59.533 DEBUG:boto:auth_path=/mlvank-26/ 2022-07-18T03:42:59.534 DEBUG:boto:path=/mlvank-26/?versioning 2022-07-18T03:42:59.535 DEBUG:boto:auth_path=/mlvank-26/?versioning 2022-07-18T03:42:59.536 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.536 DEBUG:boto:Path: /mlvank-26/?versioning 2022-07-18T03:42:59.537 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:42:59.538 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.538 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:42:59.539 DEBUG:boto:Port: 8001 2022-07-18T03:42:59.540 DEBUG:boto:Params: {} 2022-07-18T03:42:59.540 DEBUG:boto:Token: None 2022-07-18T03:42:59.541 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-26/?versioning 2022-07-18T03:42:59.541 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:aQ90IVMMnSBPbVsdzrCgIbK6GI4= 2022-07-18T03:42:59.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:aQ90IVMMnSBPbVsdzrCgIbK6GI4=', 'Content-Length': '228'} 2022-07-18T03:42:59.549 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e678112c0f823941-0062d4d6c3-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.550 DEBUG:boto:path=/mlvank-26/ 2022-07-18T03:42:59.551 DEBUG:boto:auth_path=/mlvank-26/ 2022-07-18T03:42:59.551 DEBUG:boto:path=/mlvank-26/?versioning 2022-07-18T03:42:59.552 DEBUG:boto:auth_path=/mlvank-26/?versioning 2022-07-18T03:42:59.552 DEBUG:boto:Method: PUT 2022-07-18T03:42:59.553 DEBUG:boto:Path: /mlvank-26/?versioning 2022-07-18T03:42:59.553 DEBUG:boto:Data: Suspended Disabled 2022-07-18T03:42:59.554 DEBUG:boto:Headers: {} 2022-07-18T03:42:59.554 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:42:59.555 DEBUG:boto:Port: 8001 2022-07-18T03:42:59.555 DEBUG:boto:Params: {} 2022-07-18T03:42:59.556 DEBUG:boto:Token: None 2022-07-18T03:42:59.556 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:42:59 GMT /mlvank-26/?versioning 2022-07-18T03:42:59.557 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:aQ90IVMMnSBPbVsdzrCgIbK6GI4= 2022-07-18T03:42:59.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:42:59 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:aQ90IVMMnSBPbVsdzrCgIbK6GI4=', 'Content-Length': '230'} 2022-07-18T03:42:59.565 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a3d5d18bc617343-0062d4d6c3-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:42:59 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:42:59.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:59.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:59.567 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:59.622 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:59.623 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:59.699 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:42:59.735 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:42:59.736 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:42:59.736 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:59.736 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:42:59.736 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:42:59.737 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:42:59.737 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:42:59.737 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:42:59.737 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:42:59.738 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:42:59.738 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:42:59.739 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:42:59.562569Z" 2022-07-18T03:42:59.739 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:42:59.739 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:42:59.740 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:42:59.740 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:42:59.740 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:42:59.740 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:42:59.742 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:59.742 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:42:59.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:59.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:42:59.744 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:42:59.796 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:42:59.797 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:42:59.863 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:42:59.864 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:42:59.865 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:42:59.865 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:42:59.866 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:42:59.866 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:42:59.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:59.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:59.868 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:42:59.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:59.870 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:42:59.870 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:59.871 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:59.871 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:42:59.872 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:59.872 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:42:59.873 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:59.873 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:42:59.874 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:59.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:59.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:59.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:59.875 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:42:59.876 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:59.877 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:59.877 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:42:59.877 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:59.878 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:59.878 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:59.879 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:42:59.879 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:59.880 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:59.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:59.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:59.882 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:42:59.882 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:59.883 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:59.883 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.401009_175.1", 2022-07-18T03:42:59.884 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:59.884 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:59.885 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:59.885 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.401009Z", 2022-07-18T03:42:59.885 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:59.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:59.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:59.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:42:59.887 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:42:59.888 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:42:59.888 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:42:59.889 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:42:59.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:42:59.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:42:59.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:42:59.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:42:59.890 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:42:59.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:59.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:42:59.891 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:42:59.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:59.892 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:42:59.892 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:42:59.893 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:42:59.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:42:59.893 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:42:59Z" 2022-07-18T03:42:59.893 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:42:59.895 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.401009_175.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.401009Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:42:59Z" } 2022-07-18T03:42:59.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.401009_175.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.401009Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:42:59.896 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:42:59.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.401009_175.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:42:59.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658115779.562569_190.1 target=1_1658115779.401009_175.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:04.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:04.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:04.901 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:04.952 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:04.952 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:05.014 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:05.015 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:05.015 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:05.015 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:05.015 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:05.016 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:05.016 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:05.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:05.016 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:05.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:05.017 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:05.017 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:05.017 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:05.017 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:05.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:05.018 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:05.019 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:05.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:05.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:05.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:05.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:05.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:05.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:05.020 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:05.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:05.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:05.021 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:05.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:05.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:05.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:05.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:05.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:05.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:05.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:05.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:05.023 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:05.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:05.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:05.024 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:05.024 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:05.025 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:05.025 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:05.025 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:05.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:05.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:05.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:05.026 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:05.026 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:05.027 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:05.027 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:05.027 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:05.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:05.028 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:05.028 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:05.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:05.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:05.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:05.029 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:05.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:05.030 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:05.030 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:05.030 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:05.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:05.031 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:05Z" 2022-07-18T03:43:05.031 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:05.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:05Z" } 2022-07-18T03:43:05.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:05.034 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:05.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:05.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:10.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:10.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:10.039 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:10.096 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:10.096 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:10.164 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:10.165 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:10.165 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:10.165 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:10.165 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:10.166 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:10.166 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:10.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:10.166 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:10.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:10.167 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:10.167 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:10.167 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:10.168 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:10.168 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:10.168 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:10.168 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:10.169 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:10.169 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:10.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:10.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:10.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:10.170 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:10.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:10.171 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:10.171 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:10.171 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:10.171 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:10.172 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:10.172 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:10.172 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:10.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:10.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:10.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:10.173 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:10.173 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:10.174 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:10.174 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:10.174 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:10.174 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:10.175 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:10.175 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:10.175 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:10.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:10.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:10.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:10.176 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:10.176 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:10.177 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:10.177 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:10.177 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:10.177 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:10.178 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:10.178 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:10.178 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:10.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:10.179 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:10.179 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:10.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:10.179 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:10.180 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:10.180 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:10.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:10.180 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:10Z" 2022-07-18T03:43:10.181 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:10.182 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:10Z" } 2022-07-18T03:43:10.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:10.183 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:10.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:10.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:15.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:15.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:15.189 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:15.246 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:15.246 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:15.336 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:15.337 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:15.337 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:15.338 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:15.338 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:15.339 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:15.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:15.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:15.340 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:15.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:15.342 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:15.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:15.342 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:15.343 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:15.343 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:15.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:15.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:15.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:15.344 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:15.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:15.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:15.345 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:15.345 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:15.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:15.346 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:15.346 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:15.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:15.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:15.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:15.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:15.347 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:15.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:15.348 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:15.348 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:15.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:15.349 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:15.349 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:15.349 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:15.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:15.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:15.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:15.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:15.351 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:15.351 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:15.352 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:15.352 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:15.352 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:15.353 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:15.353 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:15.353 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:15.353 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:15.354 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:15.355 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:15.355 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:15.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:15.356 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:15.357 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:15.357 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:15.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:15.358 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:15Z" 2022-07-18T03:43:15.359 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:15.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:15Z" } 2022-07-18T03:43:15.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:15.363 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:15.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:15.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:20.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:20.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:20.368 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:20.423 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:20.424 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:20.485 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:20.485 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:20.486 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:20.487 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:20.487 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:20.487 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:20.487 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:20.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:20.488 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:20.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:20.489 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:20.489 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:20.489 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:20.489 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:20.490 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:20.490 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:20.490 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:20.490 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:20.491 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:20.491 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:20.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:20.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:20.493 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:20.493 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:20.493 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:20.494 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:20.494 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:20.495 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:20.495 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:20.496 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:20.496 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:20.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:20.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:20.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:20.497 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:20.498 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:20.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:20.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:20.499 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:20.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:20.499 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:20.499 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:20.500 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:20.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:20.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:20.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:20.501 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:20.501 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:20.501 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:20.502 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:20.502 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:20.502 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:20.503 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:20.503 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:20.503 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:20.503 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:20.504 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:20.504 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:20.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:20.505 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:20.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:20.505 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:20.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:20.506 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:20Z" 2022-07-18T03:43:20.506 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:20.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:20Z" } 2022-07-18T03:43:20.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:20.509 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:20.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:20.510 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:25.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:25.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:25.513 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:25.569 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:25.569 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:25.648 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:25.649 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:25.649 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:25.649 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:25.650 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:25.650 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:25.650 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:25.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:25.651 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:25.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:25.651 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:25.651 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:25.652 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:25.652 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:25.652 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:25.653 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:25.653 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:25.653 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:25.653 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:25.654 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:25.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:25.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:25.654 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:25.655 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:25.655 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:25.656 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:25.656 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:25.657 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:25.657 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:25.657 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:25.657 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:25.658 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:25.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:25.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:25.659 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:25.659 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:25.659 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:25.659 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:25.660 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:25.660 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:25.660 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:25.660 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:25.661 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:25.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:25.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:25.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:25.662 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:25.662 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:25.662 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:25.663 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:25.663 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:25.663 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:25.663 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:25.664 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:25.664 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:25.664 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:25.664 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:25.665 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:25.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:25.665 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:25.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:25.666 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:25.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:25.666 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:25Z" 2022-07-18T03:43:25.666 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:25.668 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:25Z" } 2022-07-18T03:43:25.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:25.669 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:25.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:25.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:30.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:30.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:30.674 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:30.722 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:30.723 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:30.802 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:30.802 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:30.803 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:30.803 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:30.804 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:30.804 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:30.805 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:30.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:30.806 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:30.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:30.807 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:30.808 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:30.808 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:30.809 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:30.809 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:30.809 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:30.810 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:30.810 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:30.811 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:30.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:30.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:30.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:30.812 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:30.813 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:30.813 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:30.813 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:30.814 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:30.814 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:30.815 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:30.815 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:30.815 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:30.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:30.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:30.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:30.816 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:30.817 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:30.817 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:30.817 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:30.817 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:30.818 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:30.818 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:30.818 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:30.819 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:30.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:30.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:30.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:30.820 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:30.820 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:30.820 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:30.820 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:30.821 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:30.821 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:30.822 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:30.822 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:30.822 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:30.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:30.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:30.824 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:30.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:30.825 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:30.825 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:30.825 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:30.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:30.826 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:30Z" 2022-07-18T03:43:30.826 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:30.827 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:30Z" } 2022-07-18T03:43:30.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:30.829 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:30.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:30.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:35.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:35.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:35.835 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:35.891 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:35.892 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:35.958 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:35.959 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:35.960 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:35.960 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:35.960 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:35.961 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:35.961 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:35.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:35.962 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:35.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:35.962 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:35.963 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:35.963 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:35.963 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:35.964 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:35.964 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:35.964 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:35.964 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:35.965 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:35.965 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:35.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:35.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:35.966 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:35.966 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:35.966 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:35.967 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:35.967 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:35.967 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:35.968 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:35.968 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:35.968 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:35.968 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:35.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:35.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:35.969 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:35.970 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:35.970 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:35.970 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:35.971 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:35.971 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:35.972 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:35.972 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:35.973 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:35.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:35.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:35.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:35.975 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:35.976 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:35.976 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:35.977 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:35.977 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:35.978 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:35.978 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:35.979 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:35.979 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:35.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:35.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:35.980 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:35.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:35.980 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:35.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:35.981 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:35.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:35.981 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:35Z" 2022-07-18T03:43:35.981 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:35.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:35Z" } 2022-07-18T03:43:35.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:35.985 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:35.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:35.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:40.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:40.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:40.990 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:41.046 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:41.046 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:41.122 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:41.123 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:41.123 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:41.123 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:41.124 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:41.124 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:41.124 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:41.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:41.125 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:41.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:41.125 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:41.126 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:41.126 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:41.126 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:41.127 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:41.127 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:41.127 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:41.127 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:41.128 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:41.128 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:41.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:41.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:41.129 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:41.129 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:41.130 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:41.131 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:41.131 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:41.132 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:41.132 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:41.133 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:41.134 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:41.134 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:41.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:41.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:41.135 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:41.135 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:41.136 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:41.137 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:41.137 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:41.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:41.138 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:41.139 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:41.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:41.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:41.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:41.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:41.140 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:41.140 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:41.141 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:41.141 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:41.141 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:41.141 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:41.142 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:41.142 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:41.142 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:41.143 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:41.144 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:41.144 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:41.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:41.144 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:41.145 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:41.145 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:41.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:41.145 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:41Z" 2022-07-18T03:43:41.146 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:41.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:41Z" } 2022-07-18T03:43:41.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:41.149 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:41.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:41.150 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:46.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:46.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:46.153 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:46.209 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:46.209 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:46.289 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:46.290 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:46.290 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:46.290 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:46.291 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:46.291 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:46.291 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:46.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:46.292 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:46.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:46.292 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:46.292 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:46.293 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:46.293 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:46.293 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:46.293 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:46.294 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:46.294 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:46.295 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:46.295 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:46.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:46.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:46.296 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:46.296 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:46.296 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:46.297 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:46.297 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:46.297 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:46.298 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:46.298 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:46.298 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:46.298 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:46.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:46.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:46.299 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:46.299 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:46.300 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:46.300 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:46.300 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:46.300 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:46.301 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:46.301 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:46.301 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:46.301 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:46.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:46.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:46.302 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:46.303 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:46.303 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:46.303 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:46.303 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:46.304 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:46.304 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:46.304 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:46.305 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:46.305 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:46.306 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:46.306 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:46.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:46.306 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:46.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:46.307 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:46.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:46.307 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:46Z" 2022-07-18T03:43:46.308 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:46.309 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:46Z" } 2022-07-18T03:43:46.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:46.311 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:46.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:46.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:51.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:51.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:51.318 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:51.369 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:51.369 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:51.455 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:51.456 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:51.456 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:51.457 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:51.457 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:51.458 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:51.458 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:51.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:51.460 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:51.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:51.461 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:51.462 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:51.463 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:51.463 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:51.464 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:51.464 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:51.464 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:51.464 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:51.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:51.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:51.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:51.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:51.467 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:51.467 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:51.468 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:51.468 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:51.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:51.470 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:51.470 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:51.470 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:51.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:51.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:51.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:51.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:51.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:51.472 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:51.472 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:51.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:51.474 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:51.474 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:51.475 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:51.475 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:51.475 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:51.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:51.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:51.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:51.477 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:51.478 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:51.478 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:51.479 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:51.479 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:51.479 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:51.480 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:51.480 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:51.481 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:51.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:51.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:51.482 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:51.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:51.482 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:51.482 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:51.483 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:51.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:51.483 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:51Z" 2022-07-18T03:43:51.484 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:51.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:51Z" } 2022-07-18T03:43:51.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:51.485 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:51.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:51.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:43:56.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:56.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:43:56.489 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:43:56.545 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:43:56.545 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:43:56.611 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:43:56.612 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:43:56.613 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:43:56.613 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:43:56.613 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:43:56.614 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:43:56.614 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:56.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:56.615 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:43:56.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:56.615 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:43:56.616 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:56.616 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:56.616 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:43:56.616 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:56.617 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:43:56.617 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:56.617 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:43:56.618 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:56.618 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:56.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:43:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:43:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:43:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:43:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:43:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:43:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:43:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:43:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:43:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:43:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:43:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:43:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:43:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:43:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:43:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:43:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:43:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:43:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:43:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:43:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:43:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:43:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:43:56Z" 2022-07-18T03:43:56.633 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:43:56.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:43:56Z" } 2022-07-18T03:43:56.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:43:56.636 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:43:56.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:43:56.637 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:01.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:01.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:01.641 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:01.697 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:01.697 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:01.785 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:01.785 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:01.785 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:01.786 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:01.786 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:01.786 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:01.787 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:01.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:01.787 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:01.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:01.788 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:01.788 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:01.788 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:01.788 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:01.789 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:01.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:01.789 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:01.789 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:01.790 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:01.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:01.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:01.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:01.791 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:01.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:01.791 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:01.791 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:01.792 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:01.792 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:01.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:01.793 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:01.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:01.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:01.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:01.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:01.794 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:01.794 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:01.795 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:01.795 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:01.795 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:01.796 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:01.796 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:01.796 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:01.796 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:01.797 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:01.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:01.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:01.797 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:01.798 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:01.798 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:01.798 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:01.799 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:01.799 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:01.799 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:01.799 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:01.800 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:01.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:01.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:01.800 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:01.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:01.801 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:01.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:01.801 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:01.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:01.802 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:01Z" 2022-07-18T03:44:01.802 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:01.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:01Z" } 2022-07-18T03:44:01.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:01.805 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:01.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:01.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:06.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:06.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:06.809 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:06.864 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:06.865 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:06.929 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:06.929 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:06.929 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:06.930 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:06.930 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:06.930 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:06.930 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:06.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:06.931 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:06.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:06.932 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:06.932 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:06.932 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:06.933 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:06.933 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:06.933 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:06.934 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:06.934 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:06.934 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:06.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:06.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:06.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:06.936 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:06.936 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:06.936 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:06.936 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:06.937 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:06.937 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:06.937 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:06.937 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:06.938 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:06.938 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:06.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:06.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:06.939 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:06.939 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:06.939 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:06.939 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:06.940 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:06.940 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:06.940 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:06.940 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:06.941 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:06.941 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:06.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:06.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:06.942 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:06.943 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:06.943 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:06.944 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:06.944 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:06.944 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:06.945 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:06.945 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:06.945 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:06.946 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:06.946 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:06.947 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:06.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:06.947 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:06.948 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:06.948 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:06.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:06.948 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:06Z" 2022-07-18T03:44:06.949 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:06.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:06Z" } 2022-07-18T03:44:06.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:06.951 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:06.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:06.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:11.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:11.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:11.956 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:12.009 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:12.010 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:12.094 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:12.094 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:12.095 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:12.096 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:12.096 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:12.097 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:12.097 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:12.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:12.098 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:12.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:12.098 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:12.098 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:12.099 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:12.099 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:12.099 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:12.100 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:12.100 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:12.100 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:12.101 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:12.102 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:12.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:12.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:12.104 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:12.105 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:12.106 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:12.107 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:12.107 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:12.108 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:12.109 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:12.110 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:12.110 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:12.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:12.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:12.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:12.112 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:12.113 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:12.114 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:12.114 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:12.114 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:12.114 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:12.115 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:12.115 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:12.116 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:12.116 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:12.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:12.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:12.118 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:12.119 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:12.119 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:12.120 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:12.120 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:12.121 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:12.122 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:12.122 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:12.123 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:12.123 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:12.124 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:12.124 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:12.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:12.125 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:12.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:12.125 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:12.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:12.126 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:12Z" 2022-07-18T03:44:12.126 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:12.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:12Z" } 2022-07-18T03:44:12.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:12.130 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:12.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:12.132 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:17.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:17.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:17.135 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:17.191 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:17.191 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:17.269 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:17.269 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:17.270 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:17.270 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:17.270 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:17.271 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:17.271 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:17.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:17.271 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:17.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:17.272 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:17.272 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:17.272 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:17.272 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:17.273 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:17.273 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:17.273 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:17.273 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:17.274 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:17.274 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:17.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:17.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:17.275 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:17.275 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:17.275 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:17.275 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:17.276 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:17.276 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:17.276 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:17.276 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:17.277 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:17.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:17.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:17.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:17.278 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:17.278 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:17.278 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:17.278 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:17.279 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:17.279 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:17.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:17.279 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:17.280 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:17.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:17.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:17.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:17.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:17.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:17.281 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:17.281 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:17.282 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:17.282 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:17.282 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:17.282 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:17.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:17.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:17.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:17.283 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:17.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:17.284 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:17.284 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:17.284 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:17.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:17.285 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:17Z" 2022-07-18T03:44:17.285 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:17.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:17Z" } 2022-07-18T03:44:17.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:17.288 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:17.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:17.288 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:22.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:22.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:22.291 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:22.347 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:22.348 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:22.410 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:22.411 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:22.411 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:22.412 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:22.412 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:22.412 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:22.412 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:22.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:22.413 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:22.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:22.413 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:22.414 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:22.414 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:22.414 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:22.415 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:22.415 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:22.415 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:22.416 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:22.416 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:22.416 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:22.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:22.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:22.417 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:22.417 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:22.418 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:22.418 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:22.419 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:22.419 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:22.419 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:22.419 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:22.420 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:22.420 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:22.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:22.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:22.421 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:22.421 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:22.421 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:22.422 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:22.422 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:22.422 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:22.423 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:22.423 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:22.423 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:22.423 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:22.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:22.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:22.425 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:22.425 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:22.426 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:22.426 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:22.427 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:22.427 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:22.428 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:22.428 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:22.428 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:22.428 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:22.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:22.429 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:22.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:22.430 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:22.430 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:22.430 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:22.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:22.431 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:22Z" 2022-07-18T03:44:22.431 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:22.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:22Z" } 2022-07-18T03:44:22.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:22.434 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:22.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:22.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:27.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:27.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:27.439 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:27.494 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:27.495 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:27.573 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:27.573 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:27.573 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:27.573 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:27.574 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:27.574 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:27.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:27.575 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:27.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:27.575 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:27.576 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:27.576 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:27.576 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:27.576 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:27.577 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:27.577 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:27.577 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:27.577 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:27.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:27.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:27.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:27.578 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:27.579 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:27.579 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:27.579 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:27.579 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:27.580 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:27.580 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:27.580 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:27.580 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:27.581 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:27.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:27.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:27.581 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:27.581 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:27.582 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:27.582 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:27.582 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:27.583 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:27.583 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:27.583 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:27.583 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:27.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:27.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:27.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:27.585 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:27.585 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:27.585 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:27.585 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:27.586 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:27.586 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:27.586 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:27.586 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:27.587 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:27.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:27.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:27.587 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:27.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:27.588 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:27.588 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:27.588 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:27.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:27.589 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:27Z" 2022-07-18T03:44:27.589 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:27.591 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:27Z" } 2022-07-18T03:44:27.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:27.592 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:27.592 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:27.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:32.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:32.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:32.596 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:32.648 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:32.649 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:32.735 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:32.735 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:32.736 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:32.736 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:32.736 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:32.736 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:32.737 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:32.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:32.737 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:32.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:32.738 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:32.738 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:32.738 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:32.739 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:32.739 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:32.740 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:32.740 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:32.741 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:32.741 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:32.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:32.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:32.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:32.743 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:32.743 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:32.744 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:32.744 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:32.744 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:32.745 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:32.745 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:32.745 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:32.745 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:32.746 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:32.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:32.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:32.747 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:32.747 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:32.747 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:32.747 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:32.748 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:32.748 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:32.749 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:32.749 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:32.750 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:32.750 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:32.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:32.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:32.751 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:32.751 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:32.751 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:32.752 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:32.752 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:32.752 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:32.752 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:32.753 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:32.753 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:32.754 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:32.754 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:32.755 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:32.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:32.756 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:32.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:32.757 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:32.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:32.758 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:32Z" 2022-07-18T03:44:32.758 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:32.760 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:32Z" } 2022-07-18T03:44:32.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:32.762 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:32.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:32.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:37.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:37.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:37.767 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:37.817 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:37.817 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:37.879 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:37.880 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:37.881 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:37.881 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:37.882 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:37.882 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:37.883 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:37.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:37.884 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:37.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:37.885 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:37.885 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:37.886 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:37.887 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:37.887 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:37.888 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:37.888 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:37.889 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:37.889 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:37.890 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:37.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:37.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:37.891 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:37.891 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:37.892 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:37.892 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:37.893 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:37.893 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:37.894 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:37.894 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:37.895 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:37.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:37.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:37.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:37.896 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:37.897 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:37.897 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:37.898 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:37.898 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:37.898 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:37.899 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:37.899 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:37.900 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:37.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:37.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:37.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:37.902 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:37.902 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:37.903 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:37.903 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:37.903 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:37.904 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:37.904 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:37.905 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:37.905 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:37.906 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:37.906 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:37.907 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:37.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:37.908 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:37.908 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:37.909 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:37.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:37.910 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:37Z" 2022-07-18T03:44:37.910 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:37.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:37Z" } 2022-07-18T03:44:37.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:37.914 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:37.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:37.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:42.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:42.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:42.919 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:42.972 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:42.972 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:43.048 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:43.048 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:43.048 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:43.049 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:43.049 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:43.049 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:43.050 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:43.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:43.051 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:43.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:43.052 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:43.053 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:43.053 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:43.054 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:43.054 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:43.055 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:43.055 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:43.056 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:43.056 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:43.057 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:43.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:43.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:43.059 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:43.060 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:43.060 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:43.061 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:43.061 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:43.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:43.062 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:43.063 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:43.063 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:43.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:43.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:43.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:43.064 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:43.064 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:43.064 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:43.065 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:43.065 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:43.066 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:43.066 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:43.067 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:43.067 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:43.068 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:43.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:43.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:43.069 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:43.070 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:43.071 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:43.071 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:43.072 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:43.072 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:43.073 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:43.073 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:43.074 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:43.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:43.075 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:43.075 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:43.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:43.076 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:43.077 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:43.077 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:43.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:43.078 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:43Z" 2022-07-18T03:44:43.078 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:43.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:43Z" } 2022-07-18T03:44:43.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:43.081 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:43.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:43.083 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:48.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:48.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:48.087 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:48.142 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:48.143 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:48.220 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:48Z" 2022-07-18T03:44:48.237 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:48.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:48Z" } 2022-07-18T03:44:48.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:48.240 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:48.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:48.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:53.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:53.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:53.245 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:53.301 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:53.302 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:53.381 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:53.382 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:53.382 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:53.382 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:53.383 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:53.384 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:53.384 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:53.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:53.385 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:53.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:53.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:53.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:53.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:53.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:53.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:53.389 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:53.389 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:53.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:53.390 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:53.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:53.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:53.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:53.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:53.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:53.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:53.394 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:53.394 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:53.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:53.396 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:53.396 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:53.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:53.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:53.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:53.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:53.399 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:53.400 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:53.400 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:53.401 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:53.402 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:53.403 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:53.403 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:53.404 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:53.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:53.405 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:53.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:53.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:53.407 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:53.407 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:53.408 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:53.408 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:53.408 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:53.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:53.409 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:53.409 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:53.410 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:53.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:53.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:53.411 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:53.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:53.411 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:53.411 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:53.411 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:53.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:53.412 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:53Z" 2022-07-18T03:44:53.412 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:53.413 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:53Z" } 2022-07-18T03:44:53.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:53.415 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:53.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:53.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:44:58.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:58.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:44:58.421 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:44:58.478 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:44:58.479 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:44:58.536 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:44:58.536 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:44:58.537 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:44:58.537 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:44:58.537 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:44:58.538 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:44:58.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:58.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:58.539 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:44:58.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:58.539 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:44:58.540 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:58.540 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:58.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:44:58.542 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:58.542 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:44:58.543 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:58.543 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:44:58.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:58.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:58.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:58.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:58.546 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:44:58.547 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:58.547 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:58.548 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:44:58.548 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:58.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:58.549 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:58.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:44:58.551 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:58.551 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:58.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:58.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:58.553 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:44:58.553 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:58.554 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:58.554 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:44:58.555 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:58.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:58.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:58.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:44:58.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:58.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:58.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:58.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:44:58.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:44:58.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:44:58.560 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:44:58.560 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:44:58.561 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:44:58.561 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:44:58.562 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:44:58.562 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:44:58.563 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:44:58.563 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:58.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:44:58.564 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:44:58.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:58.565 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:44:58.566 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:44:58.566 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:44:58.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:44:58.567 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:44:58Z" 2022-07-18T03:44:58.567 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:44:58.568 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:44:58Z" } 2022-07-18T03:44:58.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:44:58.569 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:44:58.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:44:58.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:03.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:03.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:03.574 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:03.630 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:03.631 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:03.701 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:03.702 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:03.703 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:03.703 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:03.704 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:03.704 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:03.705 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:03.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:03.706 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:03.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:03.707 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:03.707 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:03.708 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:03.708 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:03.708 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:03.709 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:03.709 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:03.710 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:03.710 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:03.711 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:03.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:03.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:03.712 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:03.713 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:03.713 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:03.713 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:03.714 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:03.714 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:03.714 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:03.715 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:03.715 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:03.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:03.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:03.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:03.716 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:03.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:03.717 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:03.717 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:03.717 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:03.718 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:03.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:03.718 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:03.719 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:03.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:03.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:03.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:03.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:03.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:03.723 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:03.723 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:03.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:03.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:03.724 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:03.724 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:03.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:03.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:03.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:03.725 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:03.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:03.726 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:03.726 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:03.727 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:03.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:03.727 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:03Z" 2022-07-18T03:45:03.727 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:03.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:03Z" } 2022-07-18T03:45:03.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:03.730 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:03.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:03.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:08.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:08.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:08.734 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:08.783 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:08.784 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:08.865 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:08Z" 2022-07-18T03:45:08.884 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:08.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:08Z" } 2022-07-18T03:45:08.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:08.887 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:08.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:08.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:13.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:13.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:13.890 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:13.946 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:13.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:14.009 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:14.010 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:14.010 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:14.010 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:14.011 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:14.011 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:14.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:14.013 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:14.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:14.014 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:14.017 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:14.017 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:14.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:14.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:14.018 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:14.018 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:14.018 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:14.019 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:14.019 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:14.019 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:14.020 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:14.020 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:14.020 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:14.021 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:14.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:14.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:14.021 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:14.022 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:14.022 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:14.022 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:14.022 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:14.023 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:14.023 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:14.024 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:14.024 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:14.024 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:14.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:14.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:14.026 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:14.026 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:14.027 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:14.028 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:14.030 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:14.030 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:14.030 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:14.031 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:14.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:14.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:14.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:14.032 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:14.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:14.032 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:14.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:14.033 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:14.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:14.033 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:13Z" 2022-07-18T03:45:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:14.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:13Z" } 2022-07-18T03:45:14.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:14.037 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:14.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:14.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:19.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:19.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:19.047 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:19.096 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:19.096 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:19.158 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:19.159 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:19.159 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:19.159 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:19.160 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:19.160 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:19.160 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:19.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:19.161 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:19.162 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:19.162 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:19.162 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:19.163 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:19.163 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:19.163 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:19.163 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:19.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:19.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:19.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:19.165 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:19.165 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:19.165 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:19.165 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:19.166 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:19.166 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:19.166 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:19.166 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:19.167 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:19.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:19.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:19.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:19.168 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:19.168 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:19.168 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:19.168 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:19.169 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:19.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:19.171 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:19.172 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:19.173 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:19.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:19.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:19.173 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:19.175 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:19.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:19.175 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:19Z" 2022-07-18T03:45:19.175 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:19.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:19Z" } 2022-07-18T03:45:19.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:19.178 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:19.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:19.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:24.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:24.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:24.182 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:24.238 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:24.239 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:24.312 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:24.312 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:24.313 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:24.314 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:24.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:24.315 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:24.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:24.315 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:24.315 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:24.316 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:24.317 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:24.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:24.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:24.318 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:24.318 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:24.319 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:24.319 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:24.320 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:24.320 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:24.320 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:24.321 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:24.321 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:24.321 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:24.322 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:24.323 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:24.323 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:24.323 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:24.324 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:24.324 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:24.324 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:24.324 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:24.325 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:24.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:24.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:24.326 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:24.326 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:24.326 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:24.326 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:24.327 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:24.327 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:24.327 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:24.330 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:24.331 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:24.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:24.331 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:24Z" 2022-07-18T03:45:24.331 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:24.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:24Z" } 2022-07-18T03:45:24.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:24.334 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:24.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:24.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:29.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:29.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:29.337 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:29.417 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:29.418 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:29.544 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:29.545 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:29.545 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:29.546 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:29.546 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:29.547 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:29.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:29.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:29.548 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:29.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:29.549 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:29.550 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:29.550 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:29.551 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:29.551 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:29.552 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:29.552 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:29.552 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:29.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:29.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:29.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:29.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:29.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:29.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:29.555 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:29.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:29.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:29.556 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:29.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:29.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:29.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:29.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:29.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:29.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:29.558 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:29.558 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:29.558 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:29.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:29.559 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:29.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:29.559 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:29.560 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:29.560 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:29.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:29.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:29.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:29.561 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:29.561 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:29.561 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:29.562 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:29.562 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:29.562 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:29.562 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:29.563 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:29.563 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:29.563 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:29.563 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:29.564 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:29.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:29.564 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:29.564 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:29.565 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:29.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:29.565 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:29Z" 2022-07-18T03:45:29.565 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:29.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:29Z" } 2022-07-18T03:45:29.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:29.568 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:29.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:29.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:34.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:34.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:34.572 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:34.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:34.622 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:34.708 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:34.709 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:34.709 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:34.709 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:34.709 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:34.710 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:34.710 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:34.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:34.711 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:34.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:34.711 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:34.711 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:34.712 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:34.712 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:34.713 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:34.713 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:34.714 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:34.714 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:34.714 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:34.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:34.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:34.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:34.715 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:34.716 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:34.716 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:34.717 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:34.717 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:34.717 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:34.718 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:34.718 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:34.718 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:34.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:34.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:34.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:34.719 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:34.720 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:34.720 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:34.720 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:34.720 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:34.721 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:34.721 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:34.721 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:34.721 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:34.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:34.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:34.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:34.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:34.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:34.723 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:34.723 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:34.724 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:34.724 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:34.724 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:34.725 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:34.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:34.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:34.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:34.726 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:34.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:34.726 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:34.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:34.727 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:34.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:34.728 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:34Z" 2022-07-18T03:45:34.729 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:34.731 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:34Z" } 2022-07-18T03:45:34.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:34.733 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:34.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:34.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:39.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:39.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:39.739 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:39.810 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:39.811 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:39.920 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:39.921 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:39.921 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:39.921 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:39.922 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:39.923 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:39.923 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:39.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:39.924 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:39.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:39.925 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:39.925 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:39.926 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:39.926 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:39.927 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:39.927 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:39.928 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:39.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:39.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:39.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:39.930 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:39.931 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:39.942 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:39Z" 2022-07-18T03:45:39.942 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:39.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:39Z" } 2022-07-18T03:45:39.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:39.944 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:39.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:39.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:44.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:44.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:44.948 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:45.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:45.006 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:45.098 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:45.099 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:45.099 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:45.099 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:45.100 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:45.100 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:45.100 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:45.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:45.101 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:45.102 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:45.102 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:45.102 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:45.103 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:45.103 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:45.103 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:45.103 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:45.104 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:45.105 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:45.105 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:45.105 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:45.106 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:45.106 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:45.106 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:45.106 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:45.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:45.108 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:45.108 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:45.108 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:45.108 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:45.109 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:45.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:45.111 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:45.111 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:45.111 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:45Z" 2022-07-18T03:45:45.116 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:45.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:45Z" } 2022-07-18T03:45:45.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:45.118 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:45.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:45.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:50.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:50.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:50.122 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:50.178 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:50.178 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:50.241 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:50.242 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:50.242 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:50.246 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:50.246 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:50.246 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:50.247 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:50.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:50.256 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:50.256 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:50.256 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:50.256 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:50.257 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:50.257 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:50.257 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:50.257 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:50.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:50.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:50.258 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:50.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:50Z" 2022-07-18T03:45:50.260 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:50.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:50Z" } 2022-07-18T03:45:50.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:50.263 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:50.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:50.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:45:55.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:55.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:45:55.266 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:45:55.321 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:45:55.322 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:45:55.407 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:45:55.408 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:45:55.408 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:45:55.409 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:45:55.409 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:45:55.410 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:45:55.410 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:55.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:55.411 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:45:55.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:55.412 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:45:55.413 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:55.413 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:55.414 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:45:55.414 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:45:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:45:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:45:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:45:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:45:55.419 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:55.419 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:55.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:55.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:55.420 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:45:55.420 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:55.420 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:55.420 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:45:55.421 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:55.421 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:55.421 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:55.421 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:45:55.422 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:55.422 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:55.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:55.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:45:55.423 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:45:55.423 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:45:55.423 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:45:55.423 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:45:55.424 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:45:55.424 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:45:55.424 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:45:55.424 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:45:55.425 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:45:55.425 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:55.425 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:45:55.425 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:45:55.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:55.426 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:45:55.426 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:45:55.426 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:45:55.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:45:55.427 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:45:55Z" 2022-07-18T03:45:55.427 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:45:55.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:45:55Z" } 2022-07-18T03:45:55.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:45:55.430 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:45:55.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:45:55.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:00.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:00.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:00.433 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:00.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:00.492 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:00.559 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:00.559 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:00.560 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:00.560 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:00.562 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:00.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:00.563 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:00.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:00.564 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:00.564 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:00.565 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:00.565 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:00.566 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:00.566 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:00.567 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:00.567 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:00.567 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:00.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:00.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:00.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:00.569 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:00.569 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:00.570 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:00.570 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:00.571 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:00.571 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:00.571 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:00.572 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:00.572 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:00.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:00.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:00.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:00.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:00.573 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:00.574 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:00.574 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:00.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:00.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:00.575 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:00.575 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:00.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:00.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:00.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:00.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:00.577 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:00.577 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:00.577 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:00.578 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:00.578 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:00.579 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:00.579 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:00.579 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:00.580 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:00.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:00.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:00.580 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:00.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:00.581 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:00.581 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:00.581 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:00.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:00.582 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:00Z" 2022-07-18T03:46:00.582 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:00.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:00Z" } 2022-07-18T03:46:00.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:00.585 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:00.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:00.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:05.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:05.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:05.589 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:05.645 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:05.645 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:05.719 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:05.719 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:05.720 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:05.720 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:05.720 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:05.721 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:05.721 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:05.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:05.721 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:05.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:05.722 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:05.722 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:05.722 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:05.723 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:05.723 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:05.723 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:05.724 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:05.724 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:05.724 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:05.724 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:05.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:05.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:05.725 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:05.725 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:05.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:05.730 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:05.730 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:05.730 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:05.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:05.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:05.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:05.731 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:05.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:05.732 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:05.732 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:05.734 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:05.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:05.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:05.735 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:05.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:05.736 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:05.736 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:05.736 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:05.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:05.737 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:05Z" 2022-07-18T03:46:05.737 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:05.738 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:05Z" } 2022-07-18T03:46:05.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:05.740 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:05.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:05.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:10.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:10.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:10.744 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:10.803 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:10.804 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:10.877 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:10.878 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:10.878 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:10.879 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:10.879 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:10.880 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:10.880 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:10.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:10.881 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:10.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:10.882 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:10.882 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:10.883 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:10.883 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:10.884 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:10.884 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:10.884 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:10.885 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:10.885 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:10.885 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:10.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:10.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:10.887 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:10.887 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:10.888 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:10.888 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:10.889 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:10.889 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:10.890 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:10.890 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:10.891 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:10.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:10.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:10.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:10.892 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:10.893 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:10.894 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:10.894 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:10.894 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:10.895 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:10.895 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:10.895 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:10.896 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:10.896 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:10.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:10.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:10.897 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:10.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:10.898 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:10.898 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:10.899 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:10.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:10.899 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:10.899 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:10.900 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:10.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:10.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:10.901 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:10.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:10.902 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:10.903 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:10.903 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:10.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:10.904 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:10Z" 2022-07-18T03:46:10.905 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:10.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:10Z" } 2022-07-18T03:46:10.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:10.909 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:10.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:10.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:15.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:15.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:15.914 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:15.971 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:15.972 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:16.053 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:16.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:16.070 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:16.070 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:16.070 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:16.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:16.071 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:16Z" 2022-07-18T03:46:16.071 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:16.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:16Z" } 2022-07-18T03:46:16.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:16.074 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:16.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:16.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:21.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:21.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:21.078 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:21.128 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:21.128 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:21.211 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:21.211 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:21.212 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:21.212 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:21.212 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:21.212 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:21.213 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:21.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:21.213 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:21.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:21.214 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:21.214 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:21.214 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:21.215 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:21.215 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:21.215 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:21.216 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:21.216 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:21.216 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:21.216 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:21.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:21.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:21.217 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:21.218 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:21.218 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:21.218 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:21.218 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:21.219 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:21.219 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:21.219 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:21.219 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:21.220 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:21.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:21.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:21.220 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:21.221 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:21.221 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:21.221 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:21.221 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:21.222 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:21.222 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:21.222 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:21.222 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:21.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:21.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:21.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:21.227 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:21.227 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:21.227 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:21.228 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:21.228 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:21.228 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:21.228 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:21.229 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:21.229 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:21.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:21.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:21.230 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:21.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:21.231 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:21.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:21.232 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:21.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:21.232 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:21Z" 2022-07-18T03:46:21.232 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:21.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:21Z" } 2022-07-18T03:46:21.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:21.234 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:21.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:21.235 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:26.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:26.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:26.239 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:26.295 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:26.296 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:26.387 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:26.387 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:26.388 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:26.388 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:26.388 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:26.388 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:26.389 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:26.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:26.389 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:26.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:26.390 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:26.390 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:26.390 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:26.390 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:26.391 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:26.391 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:26.391 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:26.392 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:26.392 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:26.392 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:26.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:26.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:26.393 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:26.393 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:26.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:26.394 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:26.394 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:26.394 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:26.395 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:26.395 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:26.395 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:26.395 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:26.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:26.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:26.396 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:26.396 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:26.397 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:26.397 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:26.397 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:26.397 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:26.398 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:26.398 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:26.398 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:26.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:26.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:26.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:26.399 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:26.400 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:26.400 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:26.400 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:26.402 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:26.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:26.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:26.403 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:26.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:26.403 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:26.403 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:26.404 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:26.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:26.404 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:26Z" 2022-07-18T03:46:26.404 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:26.406 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:26Z" } 2022-07-18T03:46:26.407 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:26.407 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:26.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:26.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:31.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:31.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:31.411 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:31.467 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:31.467 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:31.538 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:31.539 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:31.539 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:31.540 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:31.540 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:31.541 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:31.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:31.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:31.541 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:31.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:31.542 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:31.543 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:31.543 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:31.544 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:31.544 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:31.545 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:31.545 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:31.546 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:31.546 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:31.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:31.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:31.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:31.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:31.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:31.549 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:31.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:31.550 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:31.550 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:31.551 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:31.551 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:31.552 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:31.552 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:31.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:31.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:31.554 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:31.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:31.555 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:31.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:31.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:31.556 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:31.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:31.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:31.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:31.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:31.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:31.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:31.558 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:31.558 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:31.558 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:31.558 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:31.559 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:31.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:31.559 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:31.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:31.560 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:31.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:31.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:31.561 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:31.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:31.561 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:31.561 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:31.562 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:31.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:31.562 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:31Z" 2022-07-18T03:46:31.562 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:31.564 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:31Z" } 2022-07-18T03:46:31.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:31.565 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:31.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:31.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:36.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:36.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:36.570 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:36.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:36.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:36.699 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:36.700 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:36.700 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:36.700 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:36.701 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:36.701 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:36.701 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:36.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:36.702 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:36.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:36.702 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:36.703 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:36.703 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:36.703 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:36.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:36.704 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:36.704 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:36.704 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:36.705 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:36.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:36.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:36.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:36.706 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:36.706 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:36.706 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:36.707 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:36.707 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:36.707 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:36.707 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:36.708 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:36.708 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:36.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:36.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:36.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:36.709 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:36.709 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:36.710 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:36.710 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:36.710 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:36.711 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:36.711 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:36.711 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:36.711 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:36.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:36.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:36.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:36.713 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:36.713 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:36.713 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:36.713 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:36.714 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:36.714 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:36.714 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:36.714 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:36.715 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:36.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:36.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:36.715 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:36.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:36.716 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:36.716 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:36.716 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:36.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:36.717 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:36Z" 2022-07-18T03:46:36.717 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:36.718 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:36Z" } 2022-07-18T03:46:36.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:36.720 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:36.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:36.721 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:41.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:41.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:41.724 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:41.787 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:41.787 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:41.933 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:41.934 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:41.934 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:41.935 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:41.935 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:41.936 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:41.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:41.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:41.938 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:41.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:41.939 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:41.939 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:41.940 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:41.941 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:41.941 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:41.942 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:41.943 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:41.944 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:41.944 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:41.944 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:41.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:41.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:41.945 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:41.945 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:41.946 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:41.946 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:41.946 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:41.946 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:41.947 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:41.947 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:41.947 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:41.947 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:41.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:41.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:41.948 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:41.948 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:41.949 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:41.949 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:41.949 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:41.949 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:41.950 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:41.950 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:41.950 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:41.951 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:41.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:41.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:41.952 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:41.952 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:41.952 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:41.952 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:41.953 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:41.953 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:41.953 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:41.956 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:41.957 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:41.957 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:41.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:41.959 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:41.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:41.960 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:41.960 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:41.960 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:41.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:41.961 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:41Z" 2022-07-18T03:46:41.961 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:41.963 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:41Z" } 2022-07-18T03:46:41.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:41.964 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:41.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:41.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:46.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:46.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:46.967 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:47.023 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:47.023 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:47.085 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:47.086 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:47.087 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:47.087 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:47.087 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:47.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:47.088 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:47.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:47.089 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:47.089 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:47.089 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:47.089 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:47.090 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:47.091 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:47.091 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:47.092 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:47.092 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:47.093 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:47.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:47.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:47.094 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:47.094 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:47.095 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:47.095 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:47.096 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:47.096 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:47.097 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:47.097 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:47.098 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:47.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:47.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:47.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:47.099 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:47.100 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:47.100 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:47.101 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:47.101 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:47.101 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:47.102 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:47.102 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:47.102 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:47.103 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:47.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:47.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:47.103 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:47.104 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:47.104 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:47.104 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:47.104 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:47.105 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:47.105 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:47.105 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:47.105 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:47.106 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:47.106 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:47.106 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:47.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:47.107 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:47.107 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:47.107 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:47.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:47.108 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:47Z" 2022-07-18T03:46:47.108 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:47.110 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:47Z" } 2022-07-18T03:46:47.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:47.112 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:47.112 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:47.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:52.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:52.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:52.116 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:52.172 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:52.173 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:52.248 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:52.248 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:52.249 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:52.249 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:52.249 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:52.249 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:52.250 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:52.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:52.250 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:52.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:52.251 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:52.251 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:52.252 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:52.252 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:52.252 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:52.252 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:52.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:52.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:52.254 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:52.255 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:52.256 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:52.256 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:52.256 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:52.257 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:52.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:52.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:52.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:52.258 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:52.258 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:52.258 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:52.258 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:52.259 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:52.259 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:52.259 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:52.260 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:52.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:52.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:52.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:52.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:52.261 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:52.261 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:52.261 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:52.262 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:52.262 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:52.262 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:52.262 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:52.263 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:52.263 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:52.263 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:52.263 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:52.264 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:52.265 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:52.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:52.265 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:52Z" 2022-07-18T03:46:52.265 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:52.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:52Z" } 2022-07-18T03:46:52.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:52.267 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:52.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:52.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:46:57.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:57.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:46:57.271 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:46:57.327 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:46:57.327 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:46:57.408 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:46:57.408 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:46:57.409 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:46:57.409 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:46:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:46:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:46:57.410 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:57.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:57.411 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:46:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:46:57.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:57.413 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:57.413 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:46:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:46:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:46:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:57.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:46:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:57.416 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:46:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:57.417 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:46:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:57.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:57.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:57.419 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:46:57.419 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:57.419 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:57.420 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:46:57.420 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:57.420 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:57.421 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:57.421 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:46:57.421 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:57.422 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:57.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:57.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:46:57.422 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:46:57.423 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:46:57.423 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:46:57.423 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:46:57.423 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:46:57.424 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:46:57.424 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:46:57.424 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:46:57.424 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:46:57.425 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:57.425 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:46:57.425 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:46:57.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:57.426 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:46:57.426 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:46:57.426 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:46:57.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:46:57.427 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:46:57Z" 2022-07-18T03:46:57.427 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:46:57.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:46:57Z" } 2022-07-18T03:46:57.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:46:57.430 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:46:57.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:46:57.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:02.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:02.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:02.434 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:02.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:02.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:02.552 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:02.553 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:02.553 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:02.554 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:02.554 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:02.554 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:02.554 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:02.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:02.555 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:02.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:02.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:02.556 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:02.556 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:02.556 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:02.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:02.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:02.557 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:02.558 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:02.558 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:02.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:02.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:02.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:02.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:02.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:02.560 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:02.560 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:02.560 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:02.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:02.561 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:02.561 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:02.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:02.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:02.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:02.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:02.562 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:02.562 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:02.563 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:02.563 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:02.563 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:02.563 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:02.564 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:02.564 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:02.564 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:02.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:02.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:02.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:02.565 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:02.565 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:02.566 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:02.566 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:02.566 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:02.567 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:02.567 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:02.567 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:02.567 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:02.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:02.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:02.568 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:02.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:02.569 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:02.569 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:02.569 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:02.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:02.570 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:02Z" 2022-07-18T03:47:02.570 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:02.571 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:02Z" } 2022-07-18T03:47:02.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:02.573 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:02.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:02.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:07.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:07.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:07.577 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:07.654 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:07.655 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:07.764 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:07.765 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:07.765 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:07.765 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:07.765 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:07.766 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:07.766 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:07.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:07.767 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:07.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:07.767 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:07.768 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:07.768 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:07.768 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:07.768 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:07.769 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:07.769 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:07.769 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:07.769 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:07.770 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:07.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:07.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:07.771 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:07.771 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:07.771 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:07.771 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:07.772 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:07.772 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:07.772 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:07.772 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:07.773 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:07.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:07.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:07.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:07.774 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:07.774 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:07.774 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:07.775 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:07.775 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:07.775 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:07.775 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:07.776 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:07.776 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:07.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:07.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:07.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:07.777 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:07.777 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:07.778 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:07.778 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:07.778 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:07.779 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:07.779 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:07.779 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:07.779 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:07.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:07.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:07.780 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:07.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:07.781 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:07.781 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:07.781 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:07.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:07.782 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:07Z" 2022-07-18T03:47:07.782 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:07.783 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:07Z" } 2022-07-18T03:47:07.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:07.785 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:07.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:07.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:12.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:12.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:12.788 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:12.844 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:12.845 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:12.914 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:12.914 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:12.915 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:12.915 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:12.915 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:12.915 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:12.916 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:12.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:12.916 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:12.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:12.917 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:12.918 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:12.918 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:12.918 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:12.918 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:12.919 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:12.919 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:12.919 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:12.919 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:12.920 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:12.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:12.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:12.921 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:12.921 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:12.921 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:12.921 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:12.922 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:12.922 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:12.922 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:12.922 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:12.923 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:12.923 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:12.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:12.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:12.925 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:12.925 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:12.926 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:12.926 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:12.926 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:12.927 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:12.927 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:12.927 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:12.927 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:12.928 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:12.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:12.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:12.928 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:12.929 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:12.929 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:12.929 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:12.929 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:12.930 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:12.930 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:12.930 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:12.930 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:12.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:12.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:12.931 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:12.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:12.932 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:12.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:12.932 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:12.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:12.933 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:12Z" 2022-07-18T03:47:12.933 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:12.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:12Z" } 2022-07-18T03:47:12.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:12.935 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:12.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:12.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:17.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:17.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:17.939 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:17.998 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:17.999 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:18.072 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:18.072 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:18.072 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:18.073 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:18.073 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:18.073 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:18.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:18.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:18.074 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:18.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:18.075 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:18.075 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:18.075 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:18.076 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:18.076 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:18.076 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:18.076 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:18.077 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:18.077 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:18.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:18.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:18.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:18.078 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:18.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:18.079 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:18.079 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:18.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:18.079 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:18.080 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:18.080 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:18.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:18.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:18.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:18.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:18.081 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:18.081 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:18.082 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:18.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:18.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:18.082 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:18.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:18.083 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:18.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:18.083 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:18.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:18.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:18.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:18.084 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:18.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:18.085 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:18.085 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:18.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:18.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:18.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:18.087 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:18.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:18.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:18.088 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:18.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:18.088 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:18.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:18.089 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:18.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:18.089 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:18Z" 2022-07-18T03:47:18.089 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:18.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:18Z" } 2022-07-18T03:47:18.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:18.092 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:18.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:18.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:23.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:23.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:23.095 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:23.151 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:23.152 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:23.219 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:23.220 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:23.220 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:23.220 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:23.220 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:23.221 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:23.221 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:23.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:23.221 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:23.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:23.222 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:23.222 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:23.223 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:23.223 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:23.223 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:23.223 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:23.224 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:23.224 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:23.224 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:23.224 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:23.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:23.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:23.225 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:23.225 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:23.225 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:23.226 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:23.226 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:23.226 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:23.226 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:23.227 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:23.227 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:23.227 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:23.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:23.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:23.228 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:23.228 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:23.228 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:23.229 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:23.229 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:23.229 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:23.229 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:23.230 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:23.230 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:23.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:23.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:23.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:23.231 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:23.231 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:23.231 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:23.232 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:23.232 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:23.232 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:23.232 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:23.233 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:23.233 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:23.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:23.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:23.234 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:23.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:23.234 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:23.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:23.234 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:23.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:23.235 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:23Z" 2022-07-18T03:47:23.235 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:23.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:23Z" } 2022-07-18T03:47:23.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:23.238 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:23.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:23.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:28.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:28.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:28.243 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:28.299 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:28.299 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:28.460 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:28.474 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:28.474 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:28.475 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:28.476 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:28.476 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:28.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:28.476 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:28Z" 2022-07-18T03:47:28.477 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:28.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:28Z" } 2022-07-18T03:47:28.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:28.480 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:28.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:28.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:33.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:33.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:33.487 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:33.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:33.544 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:33.608 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:33Z" 2022-07-18T03:47:33.633 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:33.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:33Z" } 2022-07-18T03:47:33.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:33.636 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:33.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:33.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:38.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:38.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:38.641 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:38.698 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:38.698 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:38.773 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:38.774 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:38.774 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:38.775 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:38.776 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:38.776 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:38.777 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:38.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:38.777 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:38.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:38.778 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:38.778 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:38.778 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:38.779 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:38.779 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:38.779 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:38.779 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:38.780 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:38.780 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:38.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:38.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:38.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:38.783 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:38.783 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:38.783 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:38.784 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:38.784 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:38.784 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:38.784 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:38.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:38.786 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:38.786 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:38.786 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:38.787 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:38.787 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:38.794 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:38Z" 2022-07-18T03:47:38.794 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:38.795 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:38Z" } 2022-07-18T03:47:38.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:38.796 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:38.797 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:38.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:43.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:43.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:43.801 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:43.857 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:43.858 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:43.918 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:43.933 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:43.933 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:43.934 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:43.934 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:43.936 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:43.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:43.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:43.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:43.938 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:43.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:43.938 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:43.939 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:43.939 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:43.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:43.940 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:43Z" 2022-07-18T03:47:43.940 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:43.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:43Z" } 2022-07-18T03:47:43.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:43.942 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:43.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:43.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:48.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:48.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:48.946 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:49.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:49.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:49.080 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:49.080 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:49.081 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:49.081 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:49.082 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:49.082 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:49.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:49.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:49.083 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:49.088 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:49.088 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:49.088 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:49.088 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:49.089 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:49.089 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:49.089 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:49.090 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:49.090 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:49.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:49.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:49.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:49.091 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:49.091 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:49.092 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:49.092 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:49.092 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:49.092 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:49.093 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:49.093 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:49.093 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:49.094 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:49.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:49.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:49.095 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:49.095 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:49.095 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:49.095 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:49.096 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:49.096 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:49.096 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:49.096 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:49.097 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:49.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:49.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:49.097 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:49.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:49.098 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:49.098 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:49.099 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:49.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:49.099 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:49Z" 2022-07-18T03:47:49.099 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:49.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:49Z" } 2022-07-18T03:47:49.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:49.102 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:49.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:49.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:54.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:54.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:54.106 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:54.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:54.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:54.220 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:54.221 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:54.221 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:54.221 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:54.222 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:54.222 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:54.222 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:54.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:54.223 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:54.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:54.223 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:54.223 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:54.224 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:54.224 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:54.224 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:54.224 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:54.225 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:54.225 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:54.225 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:54.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:54.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:54.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:54.226 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:54.226 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:54.227 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:54.227 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:54.227 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:54.227 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:54.228 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:54.228 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:54.228 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:54.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:54.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:54.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:54.230 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:54.231 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:54.231 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:54.232 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:54.232 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:54.233 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:54.233 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:54.234 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:54.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:54.236 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:54.237 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:54.238 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:54.238 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:54.238 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:54.239 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:54.239 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:54Z" 2022-07-18T03:47:54.243 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:54.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:54Z" } 2022-07-18T03:47:54.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:54.245 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:54.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:54.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:47:59.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:59.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:47:59.249 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:47:59.303 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:47:59.303 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:47:59.380 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:47:59.380 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:47:59.381 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:47:59.381 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:47:59.381 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:47:59.381 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:47:59.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:59.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:59.382 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:47:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:47:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:59.383 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:47:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:47:59.384 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:47:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:47:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:59.386 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:47:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:59.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:47:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:59.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:47:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:59.389 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:47:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:59.390 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:59.391 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:47:59.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:59.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:47:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:47:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:47:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:47:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:47:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:47:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:47:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:47:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:47:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:47:59.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:47:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:47:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:59.395 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:47:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:47:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:47:59.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:47:59.397 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:47:59Z" 2022-07-18T03:47:59.397 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:47:59.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:47:59Z" } 2022-07-18T03:47:59.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:47:59.399 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:47:59.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:47:59.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:04.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:04.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:04.403 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:04.458 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:04.458 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:04.518 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:04.519 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:04.519 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:04.520 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:04.520 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:04.521 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:04.521 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:04.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:04.522 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:04.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:04.523 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:04.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:04.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:04.525 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:04.525 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:04.526 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:04.526 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:04.527 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:04.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:04.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:04.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:04.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:04.529 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:04.530 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:04.530 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:04.531 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:04.531 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:04.532 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:04.532 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:04.532 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:04.533 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:04.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:04.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:04.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:04.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:04.535 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:04.535 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:04.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658115779.562569_190.1", 2022-07-18T03:48:04.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:04.537 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:04.537 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:04.538 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:42:59.562569Z", 2022-07-18T03:48:04.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:04.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:04.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:04.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:04.540 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:04.540 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:04.541 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:04.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:04.542 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:04.542 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:04.543 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:04.543 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:04.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:04.545 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:04.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:04.546 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:04.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:04.547 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:04Z" 2022-07-18T03:48:04.548 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:04.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658115779.562569_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:42:59.562569Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:04Z" } 2022-07-18T03:48:04.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658115779.562569_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:42:59.562569Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:04.551 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:04.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658115779.562569_190.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:04.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:09.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... FAIL 2022-07-18T03:48:09.576 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-27 2022-07-18T03:48:09.577 DEBUG:boto:path=/mlvank-27/ 2022-07-18T03:48:09.577 DEBUG:boto:auth_path=/mlvank-27/ 2022-07-18T03:48:09.578 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.579 DEBUG:boto:Path: /mlvank-27/ 2022-07-18T03:48:09.579 DEBUG:boto:Data: 2022-07-18T03:48:09.580 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.580 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.581 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.581 DEBUG:boto:Params: {} 2022-07-18T03:48:09.583 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:48:09.584 DEBUG:boto:Token: None 2022-07-18T03:48:09.585 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-27/ 2022-07-18T03:48:09.586 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:iiUSF2gau6slJSIGIL+ZQwlVPU4= 2022-07-18T03:48:09.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:iiUSF2gau6slJSIGIL+ZQwlVPU4=', 'Content-Length': '0'} 2022-07-18T03:48:09.608 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b94d21b0888bd9c7-0062d4d7f9-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.633 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-28 2022-07-18T03:48:09.633 DEBUG:boto:path=/mlvank-28/ 2022-07-18T03:48:09.634 DEBUG:boto:auth_path=/mlvank-28/ 2022-07-18T03:48:09.635 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.635 DEBUG:boto:Path: /mlvank-28/ 2022-07-18T03:48:09.636 DEBUG:boto:Data: 2022-07-18T03:48:09.636 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.637 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:09.637 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.638 DEBUG:boto:Params: {} 2022-07-18T03:48:09.639 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:48:09.639 DEBUG:boto:Token: None 2022-07-18T03:48:09.640 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-28/ 2022-07-18T03:48:09.641 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:9ucJa5hieA5o65Q2cAaLCWcmomU= 2022-07-18T03:48:09.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:9ucJa5hieA5o65Q2cAaLCWcmomU=', 'Content-Length': '0'} 2022-07-18T03:48:09.685 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000360805b0b93bf2e2-0062d4d7f9-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.685 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-29 2022-07-18T03:48:09.686 DEBUG:boto:path=/mlvank-29/ 2022-07-18T03:48:09.687 DEBUG:boto:auth_path=/mlvank-29/ 2022-07-18T03:48:09.687 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.688 DEBUG:boto:Path: /mlvank-29/ 2022-07-18T03:48:09.688 DEBUG:boto:Data: 2022-07-18T03:48:09.689 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.689 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:09.690 DEBUG:boto:Port: 8001 2022-07-18T03:48:09.690 DEBUG:boto:Params: {} 2022-07-18T03:48:09.691 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:48:09.691 DEBUG:boto:Token: None 2022-07-18T03:48:09.692 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-29/ 2022-07-18T03:48:09.693 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:c3QoPw2Nei82nPjz3tMz3lnYJbo= 2022-07-18T03:48:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:c3QoPw2Nei82nPjz3tMz3lnYJbo=', 'Content-Length': '0'} 2022-07-18T03:48:09.737 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d670b51d2945df7a-0062d4d7f9-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.738 DEBUG:boto:path=/mlvank-27/ 2022-07-18T03:48:09.738 DEBUG:boto:auth_path=/mlvank-27/ 2022-07-18T03:48:09.739 DEBUG:boto:path=/mlvank-27/?versioning 2022-07-18T03:48:09.739 DEBUG:boto:auth_path=/mlvank-27/?versioning 2022-07-18T03:48:09.740 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.740 DEBUG:boto:Path: /mlvank-27/?versioning 2022-07-18T03:48:09.741 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:48:09.741 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.742 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.742 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.743 DEBUG:boto:Params: {} 2022-07-18T03:48:09.743 DEBUG:boto:Token: None 2022-07-18T03:48:09.744 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-27/?versioning 2022-07-18T03:48:09.745 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:nHMKIH+zsrQYHrjI8gnTo5vtHIc= 2022-07-18T03:48:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:nHMKIH+zsrQYHrjI8gnTo5vtHIc=', 'Content-Length': '228'} 2022-07-18T03:48:09.754 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b89709b0a5a9e2b-0062d4d7f9-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.755 DEBUG:boto:path=/mlvank-27/ 2022-07-18T03:48:09.755 DEBUG:boto:auth_path=/mlvank-27/ 2022-07-18T03:48:09.756 DEBUG:boto:path=/mlvank-27/?versioning 2022-07-18T03:48:09.757 DEBUG:boto:auth_path=/mlvank-27/?versioning 2022-07-18T03:48:09.758 DEBUG:boto:Method: GET 2022-07-18T03:48:09.758 DEBUG:boto:Path: /mlvank-27/?versioning 2022-07-18T03:48:09.759 DEBUG:boto:Data: 2022-07-18T03:48:09.759 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.760 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.760 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.760 DEBUG:boto:Params: {} 2022-07-18T03:48:09.761 DEBUG:boto:Token: None 2022-07-18T03:48:09.762 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:09 GMT /mlvank-27/?versioning 2022-07-18T03:48:09.762 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:o5WeXbZ21CuWog2RaoyZhnsIHVU= 2022-07-18T03:48:09.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:o5WeXbZ21CuWog2RaoyZhnsIHVU=', 'Content-Length': '0'} 2022-07-18T03:48:09.764 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de7d9e161838aab6-0062d4d7f9-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.765 DEBUG:boto:EnabledDisabled 2022-07-18T03:48:09.766 DEBUG:boto:path=/mlvank-28/ 2022-07-18T03:48:09.767 DEBUG:boto:auth_path=/mlvank-28/ 2022-07-18T03:48:09.767 DEBUG:boto:path=/mlvank-28/?versioning 2022-07-18T03:48:09.768 DEBUG:boto:auth_path=/mlvank-28/?versioning 2022-07-18T03:48:09.768 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.769 DEBUG:boto:Path: /mlvank-28/?versioning 2022-07-18T03:48:09.770 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:48:09.771 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.772 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:09.773 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.773 DEBUG:boto:Params: {} 2022-07-18T03:48:09.774 DEBUG:boto:Token: None 2022-07-18T03:48:09.775 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-28/?versioning 2022-07-18T03:48:09.776 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:uQNgP/lk523AKM23HRey5ppIz8w= 2022-07-18T03:48:09.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:uQNgP/lk523AKM23HRey5ppIz8w=', 'Content-Length': '228'} 2022-07-18T03:48:09.796 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ee1789cb5e54bdae-0062d4d7f9-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.797 DEBUG:boto:path=/mlvank-28/ 2022-07-18T03:48:09.797 DEBUG:boto:auth_path=/mlvank-28/ 2022-07-18T03:48:09.798 DEBUG:boto:path=/mlvank-28/?versioning 2022-07-18T03:48:09.798 DEBUG:boto:auth_path=/mlvank-28/?versioning 2022-07-18T03:48:09.799 DEBUG:boto:Method: GET 2022-07-18T03:48:09.799 DEBUG:boto:Path: /mlvank-28/?versioning 2022-07-18T03:48:09.800 DEBUG:boto:Data: 2022-07-18T03:48:09.800 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.801 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:09.801 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.802 DEBUG:boto:Params: {} 2022-07-18T03:48:09.802 DEBUG:boto:Token: None 2022-07-18T03:48:09.803 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:09 GMT /mlvank-28/?versioning 2022-07-18T03:48:09.803 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:TFl/y5dH+2sjQ357KlFAYnfH5Ic= 2022-07-18T03:48:09.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:TFl/y5dH+2sjQ357KlFAYnfH5Ic=', 'Content-Length': '0'} 2022-07-18T03:48:09.806 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9a4f9dff5919c8b-0062d4d7f9-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.807 DEBUG:boto:EnabledDisabled 2022-07-18T03:48:09.807 DEBUG:boto:path=/mlvank-29/ 2022-07-18T03:48:09.808 DEBUG:boto:auth_path=/mlvank-29/ 2022-07-18T03:48:09.808 DEBUG:boto:path=/mlvank-29/?versioning 2022-07-18T03:48:09.809 DEBUG:boto:auth_path=/mlvank-29/?versioning 2022-07-18T03:48:09.810 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.810 DEBUG:boto:Path: /mlvank-29/?versioning 2022-07-18T03:48:09.811 DEBUG:boto:Data: Enabled Disabled 2022-07-18T03:48:09.811 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.812 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:09.812 DEBUG:boto:Port: 8001 2022-07-18T03:48:09.813 DEBUG:boto:Params: {} 2022-07-18T03:48:09.813 DEBUG:boto:Token: None 2022-07-18T03:48:09.814 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-29/?versioning 2022-07-18T03:48:09.815 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:jEGQB42R8pocDevgKMb/BNEPmSs= 2022-07-18T03:48:09.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:jEGQB42R8pocDevgKMb/BNEPmSs=', 'Content-Length': '228'} 2022-07-18T03:48:09.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004cab79b768db6ac8-0062d4d7f9-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.825 DEBUG:boto:path=/mlvank-29/ 2022-07-18T03:48:09.826 DEBUG:boto:auth_path=/mlvank-29/ 2022-07-18T03:48:09.826 DEBUG:boto:path=/mlvank-29/?versioning 2022-07-18T03:48:09.827 DEBUG:boto:auth_path=/mlvank-29/?versioning 2022-07-18T03:48:09.827 DEBUG:boto:Method: GET 2022-07-18T03:48:09.828 DEBUG:boto:Path: /mlvank-29/?versioning 2022-07-18T03:48:09.828 DEBUG:boto:Data: 2022-07-18T03:48:09.829 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.829 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:09.830 DEBUG:boto:Port: 8001 2022-07-18T03:48:09.830 DEBUG:boto:Params: {} 2022-07-18T03:48:09.831 DEBUG:boto:Token: None 2022-07-18T03:48:09.832 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:09 GMT /mlvank-29/?versioning 2022-07-18T03:48:09.832 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:mVGhHwPE65UTGokZtdEOORhRiu8= 2022-07-18T03:48:09.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:mVGhHwPE65UTGokZtdEOORhRiu8=', 'Content-Length': '0'} 2022-07-18T03:48:09.835 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa2d510f81982fbb-0062d4d7f9-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.836 DEBUG:boto:EnabledDisabled 2022-07-18T03:48:09.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2022-07-18T03:48:09.838 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-30 2022-07-18T03:48:09.838 DEBUG:boto:path=/mlvank-30/ 2022-07-18T03:48:09.839 DEBUG:boto:auth_path=/mlvank-30/ 2022-07-18T03:48:09.839 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.840 DEBUG:boto:Path: /mlvank-30/ 2022-07-18T03:48:09.840 DEBUG:boto:Data: 2022-07-18T03:48:09.841 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.841 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.842 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.842 DEBUG:boto:Params: {} 2022-07-18T03:48:09.843 DEBUG:boto:Token: None 2022-07-18T03:48:09.843 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-30/ 2022-07-18T03:48:09.844 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:c0kRsRTI8bV0NAKgcNYJyepknvI= 2022-07-18T03:48:09.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:c0kRsRTI8bV0NAKgcNYJyepknvI=', 'Content-Length': '0'} 2022-07-18T03:48:09.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000b527b0a7ace3dcc-0062d4d7f9-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.865 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-31 2022-07-18T03:48:09.866 DEBUG:boto:path=/mlvank-31/ 2022-07-18T03:48:09.866 DEBUG:boto:auth_path=/mlvank-31/ 2022-07-18T03:48:09.867 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.867 DEBUG:boto:Path: /mlvank-31/ 2022-07-18T03:48:09.868 DEBUG:boto:Data: 2022-07-18T03:48:09.868 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.869 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:09.869 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.870 DEBUG:boto:Params: {} 2022-07-18T03:48:09.870 DEBUG:boto:Token: None 2022-07-18T03:48:09.871 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-31/ 2022-07-18T03:48:09.871 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:P6pUAVZ0L9G3KWFQBQXD9Id/CTQ= 2022-07-18T03:48:09.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:P6pUAVZ0L9G3KWFQBQXD9Id/CTQ=', 'Content-Length': '0'} 2022-07-18T03:48:09.899 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b03cf2d6a1e4b777-0062d4d7f9-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.900 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-32 2022-07-18T03:48:09.901 DEBUG:boto:path=/mlvank-32/ 2022-07-18T03:48:09.901 DEBUG:boto:auth_path=/mlvank-32/ 2022-07-18T03:48:09.902 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.903 DEBUG:boto:Path: /mlvank-32/ 2022-07-18T03:48:09.903 DEBUG:boto:Data: 2022-07-18T03:48:09.904 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.904 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:09.905 DEBUG:boto:Port: 8001 2022-07-18T03:48:09.906 DEBUG:boto:Params: {} 2022-07-18T03:48:09.907 DEBUG:boto:Token: None 2022-07-18T03:48:09.907 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT /mlvank-32/ 2022-07-18T03:48:09.909 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:l6BK/KoqWorkFuditn3xHiXaD2c= 2022-07-18T03:48:09.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:l6BK/KoqWorkFuditn3xHiXaD2c=', 'Content-Length': '0'} 2022-07-18T03:48:09.935 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d665128fb1ba7a07-0062d4d7f9-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.935 DEBUG:boto:path=/mlvank-30/ 2022-07-18T03:48:09.936 DEBUG:boto:auth_path=/mlvank-30/ 2022-07-18T03:48:09.936 DEBUG:boto:path=/mlvank-30/?acl 2022-07-18T03:48:09.937 DEBUG:boto:auth_path=/mlvank-30/?acl 2022-07-18T03:48:09.937 DEBUG:boto:Method: GET 2022-07-18T03:48:09.938 DEBUG:boto:Path: /mlvank-30/?acl 2022-07-18T03:48:09.938 DEBUG:boto:Data: 2022-07-18T03:48:09.939 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.939 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.940 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.940 DEBUG:boto:Params: {} 2022-07-18T03:48:09.941 DEBUG:boto:Token: None 2022-07-18T03:48:09.942 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:09 GMT /mlvank-30/?acl 2022-07-18T03:48:09.942 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:tci5QyfTCZ/Bqs88izJW+/WPZnQ= 2022-07-18T03:48:09.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:tci5QyfTCZ/Bqs88izJW+/WPZnQ=', 'Content-Length': '0'} 2022-07-18T03:48:09.945 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e204be21c2cf2a7-0062d4d7f9-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.949 DEBUG:boto:path=/mlvank-30/ 2022-07-18T03:48:09.949 DEBUG:boto:auth_path=/mlvank-30/ 2022-07-18T03:48:09.950 DEBUG:boto:path=/mlvank-30/?acl 2022-07-18T03:48:09.950 DEBUG:boto:auth_path=/mlvank-30/?acl 2022-07-18T03:48:09.951 DEBUG:boto:Method: PUT 2022-07-18T03:48:09.951 DEBUG:boto:Path: /mlvank-30/?acl 2022-07-18T03:48:09.952 DEBUG:boto:Data: 2022-07-18T03:48:09.953 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-07-18T03:48:09.953 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.954 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.955 DEBUG:boto:Params: {} 2022-07-18T03:48:09.955 DEBUG:boto:Token: None 2022-07-18T03:48:09.956 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:09 GMT x-amz-acl:public-read /mlvank-30/?acl 2022-07-18T03:48:09.956 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:FPZVuGyWnAwG0gGigphwy0yDfng= 2022-07-18T03:48:09.957 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:FPZVuGyWnAwG0gGigphwy0yDfng=', 'Content-Length': '0'} 2022-07-18T03:48:09.977 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000082f14752d1402733-0062d4d7f9-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.978 DEBUG:boto:path=/mlvank-30/ 2022-07-18T03:48:09.979 DEBUG:boto:auth_path=/mlvank-30/ 2022-07-18T03:48:09.979 DEBUG:boto:path=/mlvank-30/?acl 2022-07-18T03:48:09.980 DEBUG:boto:auth_path=/mlvank-30/?acl 2022-07-18T03:48:09.981 DEBUG:boto:Method: GET 2022-07-18T03:48:09.982 DEBUG:boto:Path: /mlvank-30/?acl 2022-07-18T03:48:09.983 DEBUG:boto:Data: 2022-07-18T03:48:09.983 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.984 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:09.984 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.985 DEBUG:boto:Params: {} 2022-07-18T03:48:09.985 DEBUG:boto:Token: None 2022-07-18T03:48:09.986 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:09 GMT /mlvank-30/?acl 2022-07-18T03:48:09.987 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:tci5QyfTCZ/Bqs88izJW+/WPZnQ= 2022-07-18T03:48:09.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:tci5QyfTCZ/Bqs88izJW+/WPZnQ=', 'Content-Length': '0'} 2022-07-18T03:48:09.989 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9ef596e99f620f0-0062d4d7f9-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Mon, 18 Jul 2022 03:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:09.990 DEBUG:boto:path=/mlvank-31/ 2022-07-18T03:48:09.990 DEBUG:boto:auth_path=/mlvank-31/ 2022-07-18T03:48:09.991 DEBUG:boto:path=/mlvank-31/?acl 2022-07-18T03:48:09.991 DEBUG:boto:auth_path=/mlvank-31/?acl 2022-07-18T03:48:09.992 DEBUG:boto:Method: GET 2022-07-18T03:48:09.992 DEBUG:boto:Path: /mlvank-31/?acl 2022-07-18T03:48:09.993 DEBUG:boto:Data: 2022-07-18T03:48:09.994 DEBUG:boto:Headers: {} 2022-07-18T03:48:09.994 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:09.995 DEBUG:boto:Port: 8000 2022-07-18T03:48:09.996 DEBUG:boto:Params: {} 2022-07-18T03:48:09.997 DEBUG:boto:Token: None 2022-07-18T03:48:09.999 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:09 GMT /mlvank-31/?acl 2022-07-18T03:48:09.999 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:lxsTDGui5K4A020Ad98t8/pqUlA= 2022-07-18T03:48:10.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:lxsTDGui5K4A020Ad98t8/pqUlA=', 'Content-Length': '0'} 2022-07-18T03:48:10.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ae7bd7cbcf5d582-0062d4d7fa-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.002 DEBUG:boto:path=/mlvank-31/ 2022-07-18T03:48:10.003 DEBUG:boto:auth_path=/mlvank-31/ 2022-07-18T03:48:10.003 DEBUG:boto:path=/mlvank-31/?acl 2022-07-18T03:48:10.004 DEBUG:boto:auth_path=/mlvank-31/?acl 2022-07-18T03:48:10.004 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.005 DEBUG:boto:Path: /mlvank-31/?acl 2022-07-18T03:48:10.005 DEBUG:boto:Data: 2022-07-18T03:48:10.006 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-07-18T03:48:10.006 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:10.007 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.007 DEBUG:boto:Params: {} 2022-07-18T03:48:10.008 DEBUG:boto:Token: None 2022-07-18T03:48:10.008 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:10 GMT x-amz-acl:public-read /mlvank-31/?acl 2022-07-18T03:48:10.009 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:bsvs4oFTGy+/YEDTwVYjbCjD0RE= 2022-07-18T03:48:10.009 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:bsvs4oFTGy+/YEDTwVYjbCjD0RE=', 'Content-Length': '0'} 2022-07-18T03:48:10.036 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dc87aac42b9d5cff-0062d4d7fa-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.037 DEBUG:boto:path=/mlvank-31/ 2022-07-18T03:48:10.037 DEBUG:boto:auth_path=/mlvank-31/ 2022-07-18T03:48:10.038 DEBUG:boto:path=/mlvank-31/?acl 2022-07-18T03:48:10.038 DEBUG:boto:auth_path=/mlvank-31/?acl 2022-07-18T03:48:10.039 DEBUG:boto:Method: GET 2022-07-18T03:48:10.039 DEBUG:boto:Path: /mlvank-31/?acl 2022-07-18T03:48:10.040 DEBUG:boto:Data: 2022-07-18T03:48:10.040 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.040 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:10.041 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.041 DEBUG:boto:Params: {} 2022-07-18T03:48:10.042 DEBUG:boto:Token: None 2022-07-18T03:48:10.043 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:10 GMT /mlvank-31/?acl 2022-07-18T03:48:10.043 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:80QrMvcyUBLkcTm4KMXMukSQBQw= 2022-07-18T03:48:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:80QrMvcyUBLkcTm4KMXMukSQBQw=', 'Content-Length': '0'} 2022-07-18T03:48:10.045 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8fa15cb5173e831-0062d4d7fa-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.086 DEBUG:boto:path=/mlvank-32/ 2022-07-18T03:48:10.087 DEBUG:boto:auth_path=/mlvank-32/ 2022-07-18T03:48:10.087 DEBUG:boto:path=/mlvank-32/?acl 2022-07-18T03:48:10.088 DEBUG:boto:auth_path=/mlvank-32/?acl 2022-07-18T03:48:10.088 DEBUG:boto:Method: GET 2022-07-18T03:48:10.089 DEBUG:boto:Path: /mlvank-32/?acl 2022-07-18T03:48:10.089 DEBUG:boto:Data: 2022-07-18T03:48:10.090 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.090 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:10.091 DEBUG:boto:Port: 8001 2022-07-18T03:48:10.091 DEBUG:boto:Params: {} 2022-07-18T03:48:10.092 DEBUG:boto:Token: None 2022-07-18T03:48:10.092 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:10 GMT /mlvank-32/?acl 2022-07-18T03:48:10.093 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:IvzLWT+oAADZKJHZVDyBv3Xnb/s= 2022-07-18T03:48:10.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:IvzLWT+oAADZKJHZVDyBv3Xnb/s=', 'Content-Length': '0'} 2022-07-18T03:48:10.095 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078b3de766ed6b5ce-0062d4d7fa-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.096 DEBUG:boto:path=/mlvank-32/ 2022-07-18T03:48:10.096 DEBUG:boto:auth_path=/mlvank-32/ 2022-07-18T03:48:10.097 DEBUG:boto:path=/mlvank-32/?acl 2022-07-18T03:48:10.097 DEBUG:boto:auth_path=/mlvank-32/?acl 2022-07-18T03:48:10.098 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.098 DEBUG:boto:Path: /mlvank-32/?acl 2022-07-18T03:48:10.099 DEBUG:boto:Data: 2022-07-18T03:48:10.099 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-07-18T03:48:10.100 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:10.100 DEBUG:boto:Port: 8001 2022-07-18T03:48:10.101 DEBUG:boto:Params: {} 2022-07-18T03:48:10.101 DEBUG:boto:Token: None 2022-07-18T03:48:10.102 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:10 GMT x-amz-acl:public-read /mlvank-32/?acl 2022-07-18T03:48:10.103 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LaBmfVu3Qxfum+xejhHPSCUFs1M= 2022-07-18T03:48:10.103 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LaBmfVu3Qxfum+xejhHPSCUFs1M=', 'Content-Length': '0'} 2022-07-18T03:48:10.111 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000004083477dd3896d3-0062d4d7fa-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.112 DEBUG:boto:path=/mlvank-32/ 2022-07-18T03:48:10.113 DEBUG:boto:auth_path=/mlvank-32/ 2022-07-18T03:48:10.113 DEBUG:boto:path=/mlvank-32/?acl 2022-07-18T03:48:10.114 DEBUG:boto:auth_path=/mlvank-32/?acl 2022-07-18T03:48:10.114 DEBUG:boto:Method: GET 2022-07-18T03:48:10.115 DEBUG:boto:Path: /mlvank-32/?acl 2022-07-18T03:48:10.115 DEBUG:boto:Data: 2022-07-18T03:48:10.116 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.116 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:10.116 DEBUG:boto:Port: 8001 2022-07-18T03:48:10.117 DEBUG:boto:Params: {} 2022-07-18T03:48:10.118 DEBUG:boto:Token: None 2022-07-18T03:48:10.118 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:10 GMT /mlvank-32/?acl 2022-07-18T03:48:10.119 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:IvzLWT+oAADZKJHZVDyBv3Xnb/s= 2022-07-18T03:48:10.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:IvzLWT+oAADZKJHZVDyBv3Xnb/s=', 'Content-Length': '0'} 2022-07-18T03:48:10.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ad67a9f82c07460-0062d4d7fa-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2022-07-18T03:48:10.123 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-33 2022-07-18T03:48:10.123 DEBUG:boto:path=/mlvank-33/ 2022-07-18T03:48:10.124 DEBUG:boto:auth_path=/mlvank-33/ 2022-07-18T03:48:10.124 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.125 DEBUG:boto:Path: /mlvank-33/ 2022-07-18T03:48:10.125 DEBUG:boto:Data: 2022-07-18T03:48:10.126 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.126 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:10.127 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.127 DEBUG:boto:Params: {} 2022-07-18T03:48:10.128 DEBUG:boto:Token: None 2022-07-18T03:48:10.128 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:10 GMT /mlvank-33/ 2022-07-18T03:48:10.129 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:SkEZfrEBdiUDVpTpGk9+mqUaYio= 2022-07-18T03:48:10.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:SkEZfrEBdiUDVpTpGk9+mqUaYio=', 'Content-Length': '0'} 2022-07-18T03:48:10.155 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ff466d60d4748ff-0062d4d7fa-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.156 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-34 2022-07-18T03:48:10.157 DEBUG:boto:path=/mlvank-34/ 2022-07-18T03:48:10.157 DEBUG:boto:auth_path=/mlvank-34/ 2022-07-18T03:48:10.158 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.158 DEBUG:boto:Path: /mlvank-34/ 2022-07-18T03:48:10.159 DEBUG:boto:Data: 2022-07-18T03:48:10.159 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.160 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:10.160 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.161 DEBUG:boto:Params: {} 2022-07-18T03:48:10.161 DEBUG:boto:Token: None 2022-07-18T03:48:10.162 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:10 GMT /mlvank-34/ 2022-07-18T03:48:10.163 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:7OM6HvtZGYmFVMpi1s37Rjw/tBo= 2022-07-18T03:48:10.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:7OM6HvtZGYmFVMpi1s37Rjw/tBo=', 'Content-Length': '0'} 2022-07-18T03:48:10.264 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbff9c59e4e8b22a-0062d4d7fa-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.265 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-35 2022-07-18T03:48:10.266 DEBUG:boto:path=/mlvank-35/ 2022-07-18T03:48:10.266 DEBUG:boto:auth_path=/mlvank-35/ 2022-07-18T03:48:10.267 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.268 DEBUG:boto:Path: /mlvank-35/ 2022-07-18T03:48:10.268 DEBUG:boto:Data: 2022-07-18T03:48:10.269 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.269 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:10.270 DEBUG:boto:Port: 8001 2022-07-18T03:48:10.270 DEBUG:boto:Params: {} 2022-07-18T03:48:10.271 DEBUG:boto:Token: None 2022-07-18T03:48:10.271 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:10 GMT /mlvank-35/ 2022-07-18T03:48:10.272 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:tAW8aYWeZap4hsy2NKSafUHkpZA= 2022-07-18T03:48:10.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:tAW8aYWeZap4hsy2NKSafUHkpZA=', 'Content-Length': '0'} 2022-07-18T03:48:10.302 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f6d533d7732c121d-0062d4d7fa-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.303 DEBUG:boto:path=/mlvank-33/ 2022-07-18T03:48:10.303 DEBUG:boto:auth_path=/mlvank-33/ 2022-07-18T03:48:10.304 DEBUG:boto:path=/mlvank-33/?cors 2022-07-18T03:48:10.305 DEBUG:boto:auth_path=/mlvank-33/?cors 2022-07-18T03:48:10.305 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.306 DEBUG:boto:Path: /mlvank-33/?cors 2022-07-18T03:48:10.306 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-07-18T03:48:10.307 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-07-18T03:48:10.307 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:10.308 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.308 DEBUG:boto:Params: {} 2022-07-18T03:48:10.309 DEBUG:boto:Token: None 2022-07-18T03:48:10.310 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 18 Jul 2022 03:48:10 GMT /mlvank-33/?cors 2022-07-18T03:48:10.310 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:7j9rLEdQFgIiKeoIvXB3fBUD6ek= 2022-07-18T03:48:10.311 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:7j9rLEdQFgIiKeoIvXB3fBUD6ek=', 'Content-Length': '218'} 2022-07-18T03:48:10.317 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009741f94e7ba7a2e2-0062d4d7fa-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.318 DEBUG:boto:path=/mlvank-33/ 2022-07-18T03:48:10.319 DEBUG:boto:auth_path=/mlvank-33/ 2022-07-18T03:48:10.319 DEBUG:boto:path=/mlvank-33/?cors 2022-07-18T03:48:10.320 DEBUG:boto:auth_path=/mlvank-33/?cors 2022-07-18T03:48:10.321 DEBUG:boto:Method: GET 2022-07-18T03:48:10.321 DEBUG:boto:Path: /mlvank-33/?cors 2022-07-18T03:48:10.322 DEBUG:boto:Data: 2022-07-18T03:48:10.323 DEBUG:boto:Headers: {} 2022-07-18T03:48:10.323 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:10.324 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.324 DEBUG:boto:Params: {} 2022-07-18T03:48:10.325 DEBUG:boto:Token: None 2022-07-18T03:48:10.326 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:10 GMT /mlvank-33/?cors 2022-07-18T03:48:10.326 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LH2dxNi6ynw8O/F2vqvAH4A3s6k= 2022-07-18T03:48:10.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LH2dxNi6ynw8O/F2vqvAH4A3s6k=', 'Content-Length': '0'} 2022-07-18T03:48:10.328 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000062c37d5515a6671e-0062d4d7fa-10f9-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Mon, 18 Jul 2022 03:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:10.370 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-07-18T03:48:10.371 DEBUG:boto:path=/mlvank-34/ 2022-07-18T03:48:10.371 DEBUG:boto:auth_path=/mlvank-34/ 2022-07-18T03:48:10.372 DEBUG:boto:path=/mlvank-34/?cors 2022-07-18T03:48:10.372 DEBUG:boto:auth_path=/mlvank-34/?cors 2022-07-18T03:48:10.373 DEBUG:boto:Method: PUT 2022-07-18T03:48:10.373 DEBUG:boto:Path: /mlvank-34/?cors 2022-07-18T03:48:10.374 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-07-18T03:48:10.374 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-07-18T03:48:10.375 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:10.375 DEBUG:boto:Port: 8000 2022-07-18T03:48:10.376 DEBUG:boto:Params: {} 2022-07-18T03:48:10.376 DEBUG:boto:Token: None 2022-07-18T03:48:10.377 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 18 Jul 2022 03:48:10 GMT /mlvank-34/?cors 2022-07-18T03:48:10.378 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:9U1Du7yVRthAh+wmI8mLqPmrq9I= 2022-07-18T03:48:10.378 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:10 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:9U1Du7yVRthAh+wmI8mLqPmrq9I=', 'Content-Length': '218'} 2022-07-18T03:48:11.400 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000068eaf3dd8eb68a63-0062d4d7fa-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:11.401 DEBUG:boto:path=/mlvank-34/ 2022-07-18T03:48:11.402 DEBUG:boto:auth_path=/mlvank-34/ 2022-07-18T03:48:11.402 DEBUG:boto:path=/mlvank-34/?cors 2022-07-18T03:48:11.403 DEBUG:boto:auth_path=/mlvank-34/?cors 2022-07-18T03:48:11.403 DEBUG:boto:Method: GET 2022-07-18T03:48:11.404 DEBUG:boto:Path: /mlvank-34/?cors 2022-07-18T03:48:11.404 DEBUG:boto:Data: 2022-07-18T03:48:11.405 DEBUG:boto:Headers: {} 2022-07-18T03:48:11.405 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:11.406 DEBUG:boto:Port: 8000 2022-07-18T03:48:11.406 DEBUG:boto:Params: {} 2022-07-18T03:48:11.407 DEBUG:boto:Token: None 2022-07-18T03:48:11.408 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:11 GMT /mlvank-34/?cors 2022-07-18T03:48:11.408 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:o1eADx+zMGuCD5kAYETZJhMD27c= 2022-07-18T03:48:11.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:11 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:o1eADx+zMGuCD5kAYETZJhMD27c=', 'Content-Length': '0'} 2022-07-18T03:48:11.411 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a7e9dfd937312a9-0062d4d7fb-10e2-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Mon, 18 Jul 2022 03:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:11.411 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-07-18T03:48:11.412 DEBUG:boto:path=/mlvank-35/ 2022-07-18T03:48:11.413 DEBUG:boto:auth_path=/mlvank-35/ 2022-07-18T03:48:11.413 DEBUG:boto:path=/mlvank-35/?cors 2022-07-18T03:48:11.414 DEBUG:boto:auth_path=/mlvank-35/?cors 2022-07-18T03:48:11.414 DEBUG:boto:Method: PUT 2022-07-18T03:48:11.415 DEBUG:boto:Path: /mlvank-35/?cors 2022-07-18T03:48:11.415 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-07-18T03:48:11.416 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-07-18T03:48:11.416 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:11.417 DEBUG:boto:Port: 8001 2022-07-18T03:48:11.417 DEBUG:boto:Params: {} 2022-07-18T03:48:11.418 DEBUG:boto:Token: None 2022-07-18T03:48:11.419 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 18 Jul 2022 03:48:11 GMT /mlvank-35/?cors 2022-07-18T03:48:11.419 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qWS9zOuYRXN9Uqwielb/MUDCyVU= 2022-07-18T03:48:11.420 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:11 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qWS9zOuYRXN9Uqwielb/MUDCyVU=', 'Content-Length': '218'} 2022-07-18T03:48:12.455 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8039cc76a49284a-0062d4d7fb-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:12.456 DEBUG:boto:path=/mlvank-35/ 2022-07-18T03:48:12.457 DEBUG:boto:auth_path=/mlvank-35/ 2022-07-18T03:48:12.457 DEBUG:boto:path=/mlvank-35/?cors 2022-07-18T03:48:12.458 DEBUG:boto:auth_path=/mlvank-35/?cors 2022-07-18T03:48:12.458 DEBUG:boto:Method: GET 2022-07-18T03:48:12.459 DEBUG:boto:Path: /mlvank-35/?cors 2022-07-18T03:48:12.459 DEBUG:boto:Data: 2022-07-18T03:48:12.460 DEBUG:boto:Headers: {} 2022-07-18T03:48:12.460 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:12.461 DEBUG:boto:Port: 8001 2022-07-18T03:48:12.461 DEBUG:boto:Params: {} 2022-07-18T03:48:12.462 DEBUG:boto:Token: None 2022-07-18T03:48:12.462 DEBUG:boto:StringToSign: GET Mon, 18 Jul 2022 03:48:12 GMT /mlvank-35/?cors 2022-07-18T03:48:12.463 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:xKbmQ3iTYhdoYodCnv+rp/zaNLw= 2022-07-18T03:48:12.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:12 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:xKbmQ3iTYhdoYodCnv+rp/zaNLw=', 'Content-Length': '0'} 2022-07-18T03:48:12.465 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087d339695971aa36-0062d4d7fc-10f7-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Mon, 18 Jul 2022 03:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:12.510 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-07-18T03:48:12.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2022-07-18T03:48:12.512 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-36 2022-07-18T03:48:12.512 DEBUG:boto:path=/mlvank-36/ 2022-07-18T03:48:12.513 DEBUG:boto:auth_path=/mlvank-36/ 2022-07-18T03:48:12.513 DEBUG:boto:Method: PUT 2022-07-18T03:48:12.514 DEBUG:boto:Path: /mlvank-36/ 2022-07-18T03:48:12.514 DEBUG:boto:Data: 2022-07-18T03:48:12.515 DEBUG:boto:Headers: {} 2022-07-18T03:48:12.515 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:48:12.516 DEBUG:boto:Port: 8000 2022-07-18T03:48:12.516 DEBUG:boto:Params: {} 2022-07-18T03:48:12.517 DEBUG:boto:Token: None 2022-07-18T03:48:12.518 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:12 GMT /mlvank-36/ 2022-07-18T03:48:12.518 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2SG90ypDhCh3nVEa0hNxq9LHDBQ= 2022-07-18T03:48:12.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:12 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2SG90ypDhCh3nVEa0hNxq9LHDBQ=', 'Content-Length': '0'} 2022-07-18T03:48:12.540 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000046c9f782d00aa26d-0062d4d7fc-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:12.541 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-37 2022-07-18T03:48:12.541 DEBUG:boto:path=/mlvank-37/ 2022-07-18T03:48:12.542 DEBUG:boto:auth_path=/mlvank-37/ 2022-07-18T03:48:12.542 DEBUG:boto:Method: PUT 2022-07-18T03:48:12.543 DEBUG:boto:Path: /mlvank-37/ 2022-07-18T03:48:12.543 DEBUG:boto:Data: 2022-07-18T03:48:12.544 DEBUG:boto:Headers: {} 2022-07-18T03:48:12.544 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:48:12.545 DEBUG:boto:Port: 8000 2022-07-18T03:48:12.545 DEBUG:boto:Params: {} 2022-07-18T03:48:12.546 DEBUG:boto:Token: None 2022-07-18T03:48:12.546 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:12 GMT /mlvank-37/ 2022-07-18T03:48:12.547 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:C2LzScbFXfLQjlNLL00HKoeNfP4= 2022-07-18T03:48:12.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:12 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:C2LzScbFXfLQjlNLL00HKoeNfP4=', 'Content-Length': '0'} 2022-07-18T03:48:12.612 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001046bd0a672f56e1-0062d4d7fc-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:12.613 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-38 2022-07-18T03:48:12.613 DEBUG:boto:path=/mlvank-38/ 2022-07-18T03:48:12.614 DEBUG:boto:auth_path=/mlvank-38/ 2022-07-18T03:48:12.614 DEBUG:boto:Method: PUT 2022-07-18T03:48:12.615 DEBUG:boto:Path: /mlvank-38/ 2022-07-18T03:48:12.616 DEBUG:boto:Data: 2022-07-18T03:48:12.617 DEBUG:boto:Headers: {} 2022-07-18T03:48:12.618 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:48:12.619 DEBUG:boto:Port: 8001 2022-07-18T03:48:12.620 DEBUG:boto:Params: {} 2022-07-18T03:48:12.621 DEBUG:boto:Token: None 2022-07-18T03:48:12.622 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:48:12 GMT /mlvank-38/ 2022-07-18T03:48:12.623 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:qL+dg9WGkaPlsFuYJ4+Tlo9gy5M= 2022-07-18T03:48:12.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:48:12 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:qL+dg9WGkaPlsFuYJ4+Tlo9gy5M=', 'Content-Length': '0'} 2022-07-18T03:48:12.699 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a77c707b178a2d8a-0062d4d7fc-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:48:12.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:12.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:12.703 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:12.752 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:12.753 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:12.894 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:48:12.916 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:48:12.916 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:48:12.917 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:12.917 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:48:12.917 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:48:12.917 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:48:12.918 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:48:12.918 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:48:12.918 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:48:12.919 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:48:12.919 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:48:12.919 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:48:12.688198Z" 2022-07-18T03:48:12.919 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:48:12.920 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:48:12.920 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:48:12.920 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:48:12.920 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:48:12.921 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:48:12.922 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:12.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:48:12.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:12.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:12.924 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:12.986 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:12.986 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:13.053 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:13.054 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:13.055 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:13.056 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:13.057 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:13.058 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:13.059 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:13.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:13.061 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:13.062 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:13.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:13.064 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:13.065 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.596607_195.1", 2022-07-18T03:48:13.065 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:13.065 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:13.065 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:13.066 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.596607Z", 2022-07-18T03:48:13.066 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:13.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:13.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:13.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:13.067 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:13.067 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:13.067 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:13.068 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:13.068 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:13.068 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:13.068 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:13.069 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:13.069 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:13.069 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:13.069 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:13.070 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:13.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:13.070 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:13.070 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:13.071 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:13.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:13.071 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:13Z" 2022-07-18T03:48:13.072 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:13.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.596607_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.596607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:13Z" } 2022-07-18T03:48:13.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.596607_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.596607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:13.074 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:13.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.596607_195.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:13.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.596607_195.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:18.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:18.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:18.078 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:18.127 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:18.127 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:18.227 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:18.227 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:18.227 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:18.228 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:18.228 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:18.228 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:18.228 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:18.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:18.229 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:18.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:18.229 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:18.230 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:18.230 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:18.230 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:18.230 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:18.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:18.231 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:18.231 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:18.231 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:18.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:18.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:18.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:18.232 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:18.233 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:18.233 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:18.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:18.233 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:18.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:18.234 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:18.234 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:18.234 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:18.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:18.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:18.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:18.235 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:18.236 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:18.236 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:18.236 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:48:18.236 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:18.237 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:18.237 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:18.237 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:48:18.237 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:18.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:18.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:18.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:18.238 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:18.239 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:18.239 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:18.239 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:18.239 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:18.240 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:18.240 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:18.240 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:18.241 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:18.241 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:18.242 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:18.242 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:18.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:18.242 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:18.243 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:18.243 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:18.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:18.243 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:18Z" 2022-07-18T03:48:18.243 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:18.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:18Z" } 2022-07-18T03:48:18.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:18.246 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:18.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:18.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:23.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:23.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:23.250 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:23.299 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:23.299 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:23.374 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:48:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:48:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:23Z" 2022-07-18T03:48:23.391 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:23.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:23Z" } 2022-07-18T03:48:23.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:23.393 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:23.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:23.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:28.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:28.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:28.399 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:28.447 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:28.448 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:28.520 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:28.520 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:28.520 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:28.521 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:28.522 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:28.523 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:28.524 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:28.525 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:28.526 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:28.527 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:28.528 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:28.529 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:28.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:28.531 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:28.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:28.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.780521_209.1", 2022-07-18T03:48:28.534 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:28.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:28.535 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:28.535 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.780521Z", 2022-07-18T03:48:28.536 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:28.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:28.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:28.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:28.538 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:28.538 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:28.539 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:28.539 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:28.541 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:28.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:28.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:28.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:28.543 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:28.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:28.544 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:28.544 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:28.545 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:28.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:28.546 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:28Z" 2022-07-18T03:48:28.546 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:28.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.780521_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.780521Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:28Z" } 2022-07-18T03:48:28.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.780521_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.780521Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:28.550 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:28.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.780521_209.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:28.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.780521_209.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:33.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:33.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:33.555 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:33.603 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:33.604 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:33.685 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:33.685 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:33.686 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:33.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:33.688 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:33.689 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:33.690 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:33.690 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:33.690 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:33.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:33.692 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:33.693 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:33.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:33.695 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:33.695 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:33.696 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:33.696 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:48:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:33.697 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:48:33.698 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:33.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:33.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:33.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:33.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:33.699 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:33.699 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:33.699 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:33.700 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:33.700 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:33.700 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:33.700 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:33.701 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:33.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:33.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:33.701 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:33.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:33.702 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:33.702 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:33.702 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:33.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:33.703 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:33Z" 2022-07-18T03:48:33.703 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:33.704 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:33Z" } 2022-07-18T03:48:33.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:33.705 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:33.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:33.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:38.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:38.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:38.710 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:38.774 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:38.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:38.858 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:38.859 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:38.859 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:38.860 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:38.860 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:38.860 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:38.860 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:38.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:38.861 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:38.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:38.861 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:38.862 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:38.862 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:38.862 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:38.863 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:38.863 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:38.863 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:38.863 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:38.864 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:38.864 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:38.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:38.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:38.865 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:38.865 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:38.865 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:38.866 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:38.866 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:38.866 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:38.866 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:38.867 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:38.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:38.867 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:38.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:38.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:38.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:38.868 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:38.868 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:38.869 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:48:38.869 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:38.869 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:38.869 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:38.870 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:48:38.870 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:38.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:38.872 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:38.872 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:38.872 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:38.872 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:38.873 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:38.873 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:38.873 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:38.873 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:38.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:38.874 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:38.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:38.874 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:38.875 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:38.875 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:38.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:38.875 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:38Z" 2022-07-18T03:48:38.876 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:38.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:38Z" } 2022-07-18T03:48:38.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:38.878 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:38.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:38.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:43.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:43.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:43.883 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:43.936 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:43.937 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:43.998 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:44.005 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:44.005 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:44.006 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:44.006 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:44.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:44.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:44.008 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:44.008 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:44.009 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:44.009 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:44.010 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:44.010 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:44.011 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:44.011 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:44.012 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:44.013 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:44.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:44.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:44.014 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:44.015 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:44.015 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:44.015 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:48:44.016 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:44.016 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:44.017 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:44.017 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:48:44.018 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:44.018 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:44.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:44.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:44.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:44.020 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:44.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:44.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:44.022 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:44.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:44.023 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:44.023 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:44.024 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:44.024 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:44.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:44.025 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:44.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:44.026 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:44.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:44.027 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:44.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:44.028 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:43Z" 2022-07-18T03:48:44.029 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:44.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:43Z" } 2022-07-18T03:48:44.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:44.033 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:44.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:44.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:49.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:49.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:49.038 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:49.091 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:49.091 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:49.180 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.780521_209.1", 2022-07-18T03:48:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.780521Z", 2022-07-18T03:48:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:49Z" 2022-07-18T03:48:49.204 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:49.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.780521_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.780521Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:49Z" } 2022-07-18T03:48:49.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.780521_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.780521Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:49.208 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:49.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.780521_209.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:49.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.780521_209.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:54.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:54.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:54.213 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:54.262 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:54.263 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:54.322 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:54.323 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:54.323 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:54.323 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:54.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:54.325 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:54.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:54.325 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:54.325 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:54.326 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:54.326 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:54.326 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:54.326 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:54.327 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:54.327 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:54.327 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:54.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:54.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:54.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:54.328 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:54.328 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:54.329 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:54.329 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:54.329 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:54.329 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:54.330 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:54.330 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:54.330 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:54.331 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:54.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:54.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:54.332 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:54.332 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:54.332 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:54.332 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.719455_206.1", 2022-07-18T03:48:54.333 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:54.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:54.333 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:54.333 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.719455Z", 2022-07-18T03:48:54.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:54.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:54.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:54.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:54.336 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:54.336 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:54.336 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:54.337 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:54.337 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:54.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:54.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:54.338 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:54.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:54.338 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:54.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:54.339 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:54.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:54.339 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:54Z" 2022-07-18T03:48:54.339 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:54.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.719455_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.719455Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:54Z" } 2022-07-18T03:48:54.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.719455_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.719455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:54.342 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:54.342 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.719455_206.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:54.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.719455_206.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:48:59.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:59.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:48:59.346 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:48:59.395 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:48:59.396 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:48:59.468 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:48:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:48:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:48:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:48:59.469 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:48:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:48:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:59.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:59.471 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:48:59.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:59.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:48:59.472 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:59.473 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:59.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:48:59.473 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:59.474 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:48:59.474 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:59.474 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:48:59.475 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:59.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:59.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:59.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:59.477 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:48:59.477 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:59.478 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:59.478 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:48:59.478 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:59.479 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:59.479 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:59.480 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:48:59.480 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:59.480 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:59.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:59.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:59.481 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:48:59.482 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:59.482 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:59.483 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:48:59.483 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:59.483 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:59.484 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:59.484 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:48:59.484 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:59.484 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:59.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:59.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:48:59.485 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:48:59.485 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:48:59.486 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:48:59.486 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:48:59.486 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:48:59.486 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:48:59.487 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:48:59.487 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:48:59.487 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:48:59.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:59.488 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:48:59.488 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:48:59.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:59.488 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:48:59.489 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:48:59.489 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:48:59.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:48:59.489 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:48:59Z" 2022-07-18T03:48:59.490 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:48:59.493 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:48:59Z" } 2022-07-18T03:48:59.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:48:59.495 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:48:59.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:48:59.497 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:04.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:04.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:04.500 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:04.549 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:04.550 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:04.620 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:04.621 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:04.622 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:04.622 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:04.623 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:04.623 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:04.624 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:04.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:04.625 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:04.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:04.626 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:04.627 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:04.627 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:04.628 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:04.628 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:04.629 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:04.630 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:04.630 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:04.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:04.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:04.632 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:04.633 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:04.634 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:04.634 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:04.634 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:04.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:04.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:04.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:04.635 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:04.636 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:04.636 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:04.636 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.713185_205.1", 2022-07-18T03:49:04.636 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:04.637 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:04.637 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:04.637 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.713185Z", 2022-07-18T03:49:04.638 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:04.638 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:04.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:04.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:04.639 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:04.639 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:04.640 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:04.640 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:04.641 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:04.641 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:04.641 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:04.642 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:04.643 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:04.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:04.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:04.643 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:04.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:04.644 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:04.644 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:04.644 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:04.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:04.645 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:04Z" 2022-07-18T03:49:04.645 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:04.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.713185_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.713185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:04Z" } 2022-07-18T03:49:04.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.713185_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.713185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:04.648 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:04.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.713185_205.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:04.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.713185_205.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:09.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:09.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:09.652 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:09.701 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:09.701 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:09.759 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:09.759 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:09.759 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.713185_205.1", 2022-07-18T03:49:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.713185Z", 2022-07-18T03:49:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:09.771 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:09.772 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:09.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:09.774 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:09Z" 2022-07-18T03:49:09.775 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:09.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.713185_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.713185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:09Z" } 2022-07-18T03:49:09.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.713185_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.713185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:09.778 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:09.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.713185_205.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:09.779 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.713185_205.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:14.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:14.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:14.782 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:14.840 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:14.841 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:14.911 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:14.912 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:14.912 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:14.913 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:14.913 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:14.914 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:14.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:14.915 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:14.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:14.916 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:14.916 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:14.917 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:14.917 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:14.918 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:14.918 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:14.918 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:14.919 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:14.919 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:14.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:14.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:14.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:14.920 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:14.920 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:14.920 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:14.921 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:14.921 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:14.921 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:14.922 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:14.922 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:14.922 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:14.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:14.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:14.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:14.923 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:14.923 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:14.924 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:14.924 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:49:14.924 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:14.924 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:14.925 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:14.925 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:49:14.925 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:14.926 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:14.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:14.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:14.926 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:14.927 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:14.927 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:14.927 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:14.927 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:14.928 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:14.928 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:14.928 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:14.929 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:14.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:14.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:14.929 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:14.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:14.930 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:14.930 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:14.930 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:14.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:14.931 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:14Z" 2022-07-18T03:49:14.931 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:14.932 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:14Z" } 2022-07-18T03:49:14.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:14.934 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:14.934 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:14.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:19.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:19.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:19.938 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:19.996 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:19.996 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:20.059 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:20.060 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:20.060 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:20.060 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:20.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:20.062 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:20.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:20.062 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:20.063 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:20.063 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:20.063 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:20.064 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:20.064 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:20.065 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:20.065 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:20.066 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:20.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:20.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:20.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:20.068 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:20.068 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:20.069 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:20.069 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:20.070 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:20.070 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:20.071 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:20.072 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:20.072 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:20.072 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:20.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:20.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:20.074 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:20.074 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:20.075 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:20.075 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.707851_203.1", 2022-07-18T03:49:20.076 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:20.076 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:20.077 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:20.078 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.707851Z", 2022-07-18T03:49:20.078 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:20.079 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:20.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:20.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:20.080 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:20.081 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:20.081 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:20.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:20.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:20.083 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:20.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:20.084 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:20.084 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:20.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:20.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:20.086 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:20.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:20.087 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:20.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:20.088 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:20.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:20.089 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:20Z" 2022-07-18T03:49:20.090 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:20.092 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.707851_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.707851Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:20Z" } 2022-07-18T03:49:20.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.707851_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.707851Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:20.094 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:20.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.707851_203.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:20.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.707851_203.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:25.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:25.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:25.100 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:25.148 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:25.148 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:25.254 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:25.255 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:25.255 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:25.255 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:25.255 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:25.256 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:25.256 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:25.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:25.256 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:25.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:25.257 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:25.257 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:25.258 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:25.258 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:25.258 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:25.258 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:25.259 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:25.259 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:25.259 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:25.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:25.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:25.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:25.260 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:25.260 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:25.260 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:25.261 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:25.261 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:25.261 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:25.261 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:25.262 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:25.262 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:25.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:25.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:25.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:25.263 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:25.263 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:25.263 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:25.264 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:49:25.264 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:25.264 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:25.264 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:25.265 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:49:25.265 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:25.265 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:25.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:25.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:25.266 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:25.266 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:25.266 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:25.267 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:25.267 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:25.267 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:25.267 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:25.268 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:25.268 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:25.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:25.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:25.269 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:25.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:25.269 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:25.269 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:25.270 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:25.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:25.270 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:25Z" 2022-07-18T03:49:25.270 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:25.272 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:25Z" } 2022-07-18T03:49:25.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:25.273 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:25.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:25.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:30.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:30.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:30.277 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:30.326 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:30.327 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:30.391 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:30.393 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:30.393 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:30.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:49:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:49:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:30.404 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:30.405 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:30.405 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:30.406 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:30.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:30.407 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:30.407 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:30.408 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:30.408 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:30.409 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:30.409 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:30.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:30.410 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:30.411 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:30.412 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:30.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:30.412 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:30Z" 2022-07-18T03:49:30.413 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:30.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:30Z" } 2022-07-18T03:49:30.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:30.416 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:30.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:30.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:35.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:35.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:35.420 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:35.469 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:35.469 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:35.523 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:49:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:49:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:35.537 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:35.537 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:35.537 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:35.537 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:35.538 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:35.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:35.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:35.538 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:35.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:35.539 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:35.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:35.539 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:35.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:35.540 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:35Z" 2022-07-18T03:49:35.540 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:35.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:35Z" } 2022-07-18T03:49:35.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:35.542 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:35.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:35.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:40.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:40.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:40.546 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:40.600 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:40.600 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:40.689 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:40.689 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:40.689 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:40.690 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:40.690 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:40.691 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:40.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:40.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:40.692 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:40.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:40.693 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:40.694 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:40.694 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:40.695 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:40.695 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:40.696 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:40.696 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:40.697 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:40.697 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:40.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:40.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:40.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:40.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:40.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:40.700 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:40.701 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:40.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:40.702 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:40.702 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:40.703 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:40.703 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.817960_211.1", 2022-07-18T03:49:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:40.709 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.817960Z", 2022-07-18T03:49:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:40.716 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:40.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:40.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:40.718 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:40.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:40.719 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:40.719 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:40.720 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:40.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:40.721 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:40Z" 2022-07-18T03:49:40.721 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:40.724 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.817960_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.817960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:40Z" } 2022-07-18T03:49:40.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.817960_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.817960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:40.726 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:40.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.817960_211.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:40.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.817960_211.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:45.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:45.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:45.732 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:45.791 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:45.791 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:45.901 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:45.901 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:45.902 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:45.902 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:45.902 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:45.903 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:45.903 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:45.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:45.903 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:45.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:45.904 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:45.904 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:45.904 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:45.905 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:45.905 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:45.905 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:45.905 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:45.906 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:45.906 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:45.906 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:45.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:45.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:45.907 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:45.907 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:45.907 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:45.908 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:45.908 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:45.908 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:45.908 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:45.909 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:45.909 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:45.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:45.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:45.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:45.910 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:45.911 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:45.911 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:45.911 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:49:45.912 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:45.912 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:45.912 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:45.912 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:49:45.913 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:45.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:45.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:45.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:45.914 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:45.914 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:45.914 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:45.915 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:45.915 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:45.915 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:45.915 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:45.916 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:45.916 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:45.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:45.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:45.917 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:45.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:45.917 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:45.917 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:45.918 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:45.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:45.918 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:45Z" 2022-07-18T03:49:45.918 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:45.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:45Z" } 2022-07-18T03:49:45.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:45.921 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:45.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:45.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:50.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:50.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:50.925 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:50.974 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:50.974 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:51.046 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:51.046 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:51.046 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:51.047 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:51.047 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:51.048 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:51.049 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:51.050 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:51.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:51.052 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:51.053 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:51.053 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:51.053 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:51.053 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:51.054 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:51.054 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:51.054 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:51.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:51.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:51.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:51.055 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:51.056 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:51.056 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:51.056 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.719455_206.1", 2022-07-18T03:49:51.056 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:51.057 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:51.057 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:51.057 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.719455Z", 2022-07-18T03:49:51.057 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:51.058 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:51.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:51.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:51.059 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:51.059 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:51.059 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:51.059 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:51.060 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:51.060 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:51.060 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:51.060 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:51.061 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:51.061 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:51.061 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:51.061 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:51.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:51.062 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:51.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:51.062 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:51.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:51.063 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:51Z" 2022-07-18T03:49:51.063 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:51.065 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.719455_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.719455Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:51Z" } 2022-07-18T03:49:51.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.719455_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.719455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:51.066 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:51.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.719455_206.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:51.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.719455_206.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:49:56.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:56.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:49:56.070 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:49:56.120 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:49:56.120 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:49:56.187 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:49:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:49:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:49:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:49:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:49:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:49:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:49:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:49:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:49:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:49:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:49:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:49:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:49:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:49:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:49:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.707851_203.1", 2022-07-18T03:49:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.707851Z", 2022-07-18T03:49:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:49:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:49:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:49:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:49:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:49:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:49:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:49:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:49:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:49:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:49:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:49:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:49:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:49:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:49:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:49:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:49:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:49:56Z" 2022-07-18T03:49:56.206 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:49:56.207 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.707851_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.707851Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:49:56Z" } 2022-07-18T03:49:56.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.707851_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.707851Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:49:56.209 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:49:56.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.707851_203.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:49:56.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.707851_203.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:01.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:01.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:01.213 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:01.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:01.262 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:01.327 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:01.328 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:01.328 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:01.328 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:01.329 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:01.329 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:01.329 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:01.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:01.330 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:01.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:01.330 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:01.330 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:01.331 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:01.331 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:01.331 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:01.331 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:01.332 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:01.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:01.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:01.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:01.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:01.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:01.333 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:01.333 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:01.334 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:01.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:01.336 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.707851_203.1", 2022-07-18T03:50:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:01.337 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.707851Z", 2022-07-18T03:50:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:01.338 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:01.339 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:01.340 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:01.341 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:01.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:01.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:01.344 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:01Z" 2022-07-18T03:50:01.344 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:01.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.707851_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.707851Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:01Z" } 2022-07-18T03:50:01.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.707851_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.707851Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:01.356 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:01.357 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.707851_203.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:01.357 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.707851_203.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:06.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:06.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:06.364 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:06.412 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:06.413 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:06.554 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:06.555 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:06.556 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:06.556 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:06.556 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:06.557 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:06.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:06.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:06.558 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:06.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:06.559 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:06.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:06.559 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:06.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:06.560 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:06.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:06.560 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:06.560 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:06.561 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:06.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:06.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:06.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:06.562 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:06.562 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:06.562 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:06.563 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:06.563 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:06.564 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:06.564 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:06.565 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:06.565 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:06.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:06.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:06.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:06.567 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:06.568 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:06.568 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:06.569 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.707851_203.1", 2022-07-18T03:50:06.569 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:06.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:06.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:06.571 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.707851Z", 2022-07-18T03:50:06.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:06.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:06.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:06.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:06.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:06.573 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:06.574 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:06.574 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:06.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:06.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:06.576 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:06.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:06.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:06.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:06.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:06.577 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:06.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:06.578 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:06.578 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:06.578 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:06.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:06.579 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:06Z" 2022-07-18T03:50:06.579 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:06.581 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.707851_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.707851Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:06Z" } 2022-07-18T03:50:06.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.707851_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.707851Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:06.582 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:06.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.707851_203.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:06.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.707851_203.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:11.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:11.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:11.586 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:11.636 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:11.636 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:11.777 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:11.778 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:11.778 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:11.778 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:11.779 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:11.779 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:11.779 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:11.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:11.780 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:11.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:11.780 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:11.781 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:11.781 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:11.782 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:11.782 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:11.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:11.783 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:11.783 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:11.783 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:11.783 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:11.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:11.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:11.784 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:11.785 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:11.785 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:11.785 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:11.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:11.786 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:11.786 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:11.786 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:11.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:11.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:11.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:11.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:11.787 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:11.788 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:11.788 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:11.788 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:11.789 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:11.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:11.789 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:11.789 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:11.790 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:11.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:11.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:11.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:11.791 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:11.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:11.792 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:11.792 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:11.792 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:11.793 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:11.793 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:11.793 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:11.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:11.794 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:11.794 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:11.794 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:11.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:11.795 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:11.795 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:11.796 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:11.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:11.796 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:11Z" 2022-07-18T03:50:11.797 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:11.798 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:11Z" } 2022-07-18T03:50:11.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:11.799 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:11.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:11.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:16.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:16.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:16.803 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:16.852 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:16.852 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:16.944 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:16.945 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:16.945 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:16.946 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:16.946 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:16.946 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:16.947 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:16.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:16.948 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:16.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:16.948 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:16.949 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:16.949 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:16.949 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:16.950 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:16.950 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:16.951 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:16.951 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:16.951 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:16.951 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:16.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:16.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:16.952 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:16.952 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:16.953 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:16.953 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:16.953 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:16.953 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:16.954 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:16.954 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:16.954 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:16.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:16.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:16.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:16.955 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:16.956 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:16.956 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:16.956 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:16.957 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:16.957 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:16.957 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:16.957 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:16.958 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:16.958 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:16.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:16.959 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:16.960 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:16.960 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:16.960 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:16.961 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:16.961 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:16.961 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:16.962 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:16.962 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:16.962 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:16.963 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:16.964 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:16.964 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:16.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:16.965 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:16Z" 2022-07-18T03:50:16.965 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:16.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:16Z" } 2022-07-18T03:50:16.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:16.967 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:16.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:16.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:21.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:21.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:21.971 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:22.022 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:22.023 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:22.103 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:22.103 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:22.104 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:22.104 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:22.104 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:22.105 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:22.105 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:22.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:22.105 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:22.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:22.106 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:22.106 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:22.107 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:22.108 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:22.108 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:22.109 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:22.109 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:22.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:22.110 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:22.111 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:22.111 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:22.111 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:22.112 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:22.113 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:22.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:22.114 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.707851_203.1", 2022-07-18T03:50:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:22.115 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.707851Z", 2022-07-18T03:50:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:22.117 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:22.118 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:22.119 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:22.120 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:22.121 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:22.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:22.122 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:22Z" 2022-07-18T03:50:22.122 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:22.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.707851_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.707851Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:22Z" } 2022-07-18T03:50:22.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.707851_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.707851Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:22.125 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:22.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.707851_203.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:22.127 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.707851_203.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:27.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:27.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:27.130 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:27.178 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:27.179 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:27.247 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:27.248 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:27.248 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:27.249 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:27.249 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:27.249 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:27.249 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:27.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:27.250 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:27.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:27.251 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:27.251 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:27.252 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:27.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:27.254 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:27.255 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:27.256 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:27.257 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:27.258 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.713185_205.1", 2022-07-18T03:50:27.259 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:27.259 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:27.259 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.713185Z", 2022-07-18T03:50:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:27.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:27.261 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:27.262 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:27.263 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:27.264 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:27.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:27.266 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:27Z" 2022-07-18T03:50:27.266 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:27.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.713185_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.713185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:27Z" } 2022-07-18T03:50:27.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.713185_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.713185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:27.268 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:27.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.713185_205.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:27.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.713185_205.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:32.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:32.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:32.273 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:32.323 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:32.323 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:32.390 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:32.391 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:32.391 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:32.392 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:32.392 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:32.392 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:32.393 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:32.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:32.393 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:32.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:32.394 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:32.394 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:32.394 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:32.395 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:32.395 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:32.395 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:32.396 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:32.396 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:32.396 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:32.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:32.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:32.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:32.397 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:32.398 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:32.398 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:32.398 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:32.398 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:32.399 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:32.399 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:32.399 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:32.400 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:32.400 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:32.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:32.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:32.401 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:32.401 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:32.402 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:32.402 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:32.403 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:32.403 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:32.403 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:32.404 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:32.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:32.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:32.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:32.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:32.405 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:32.405 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:32.405 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:32.406 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:32.406 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:32.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:32.406 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:32.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:32.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:32.409 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:32Z" 2022-07-18T03:50:32.409 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:32.411 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:32Z" } 2022-07-18T03:50:32.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:32.412 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:32.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:32.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:37.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:37.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:37.416 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:37.465 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:37.466 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:37.542 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:37.542 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:37.543 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:37.543 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:37.543 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:37.544 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:37.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:37.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:37.544 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:37.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:37.545 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:37.545 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:37.545 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:37.546 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:37.546 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:37.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:37.546 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:37.547 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:37.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:37.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:37.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:37.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:37.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:37.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:37.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:37.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:37.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:37.550 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:37.550 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:37.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:37.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:37.551 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:37.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:37.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:37.551 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:37.552 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:37.552 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:37.552 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:37.552 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:37.553 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:37.553 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:37.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:37.554 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:37.554 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:37.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:37.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:37.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:37.555 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:37.555 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:37.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:37.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:37.556 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:37.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:37.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:37.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:37.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:37.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:37.557 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:37.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:37.558 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:37.558 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:37.559 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:37.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:37.559 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:37Z" 2022-07-18T03:50:37.559 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:37.561 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:37Z" } 2022-07-18T03:50:37.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:37.562 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:37.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:37.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:42.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:42.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:42.566 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:42.616 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:42.617 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:42.673 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:42Z" 2022-07-18T03:50:42.691 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:42.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:42Z" } 2022-07-18T03:50:42.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:42.694 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:42.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:42.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:47.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:47.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:47.698 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:47.746 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:47.747 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:47.836 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:47.838 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:47.838 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:47.838 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:47.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:47.839 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:47.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:47.840 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:47.840 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:47.840 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:47.841 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:47.841 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:47.841 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:47.841 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:47.842 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:47.842 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:47.842 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:47.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:47.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:47.843 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:47.843 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:47.844 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:47.844 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:47.844 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:47.844 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:47.845 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:47.845 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:47.845 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:47.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:47.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:47.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:47.846 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:47.847 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:47.847 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:47.847 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:47.847 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:47.848 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:47.848 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:47.848 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:47.848 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:47.849 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:47.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:47.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:47.849 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:47.850 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:47.850 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:47.850 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:47.851 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:47.851 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:47.851 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:47.852 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:47.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:47.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:47.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:47.853 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:47.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:47.853 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:47.853 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:47.854 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:47.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:47.854 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:47Z" 2022-07-18T03:50:47.854 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:47.856 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:47Z" } 2022-07-18T03:50:47.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:47.857 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:47.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:47.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:52.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:52.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:52.869 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:52.937 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:52.937 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:53.016 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:53.016 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:53.017 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:53.017 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:53.018 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:53.018 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:53.018 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:53.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:53.019 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:53.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:53.019 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:53.019 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:53.020 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:53.020 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:53.020 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:53.020 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:53.021 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:53.021 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:53.021 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:53.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:53.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:53.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:53.022 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:53.023 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:53.023 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:53.023 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:53.023 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:53.023 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:53.024 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:53.024 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:53.024 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:53.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:53.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:53.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:53.025 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:53.025 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:53.026 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:53.026 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:50:53.026 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:53.026 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:53.027 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:53.027 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:50:53.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:53.028 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:53.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:53.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:53.029 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:53.029 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:53.029 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:53.029 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:53.030 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:53.030 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:53.030 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:53.031 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:53.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:53.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:53.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:53.032 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:53.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:53.033 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:53.033 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:53.033 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:53.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:53.034 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:53Z" 2022-07-18T03:50:53.034 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:53.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:53Z" } 2022-07-18T03:50:53.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:53.036 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:53.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:53.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:50:58.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:58.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:50:58.040 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:50:58.100 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:50:58.100 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:50:58.165 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:50:58.165 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:50:58.165 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:50:58.166 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:50:58.166 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:50:58.166 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:50:58.167 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:58.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:58.168 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:50:58.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:58.169 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:50:58.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:58.170 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:58.171 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:50:58.171 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:58.172 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:50:58.172 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:58.172 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:50:58.173 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:58.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:58.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:58.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:58.174 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:50:58.174 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:58.174 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:58.175 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:50:58.175 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:58.175 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:58.175 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:58.176 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:50:58.176 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:58.176 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:58.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:58.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:58.178 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:50:58.179 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:58.179 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:58.180 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.719455_206.1", 2022-07-18T03:50:58.180 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:58.181 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:58.181 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:58.182 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.719455Z", 2022-07-18T03:50:58.182 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:58.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:58.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:58.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:50:58.183 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:50:58.183 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:50:58.184 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:50:58.184 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:50:58.185 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:50:58.185 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:50:58.186 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:50:58.186 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:50:58.187 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:50:58.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:58.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:50:58.188 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:50:58.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:58.189 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:50:58.189 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:50:58.190 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:50:58.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:50:58.191 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:50:58Z" 2022-07-18T03:50:58.191 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:50:58.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.719455_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.719455Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:50:58Z" } 2022-07-18T03:50:58.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.719455_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.719455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:50:58.194 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:50:58.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.719455_206.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:50:58.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.719455_206.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:03.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:03.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:03.198 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:03.249 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:03.249 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:03.320 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:03.321 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:03.321 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:03.321 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:03.321 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:03.322 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:03.322 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:03.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:03.323 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:03.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:03.323 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:03.323 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:03.324 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:03.324 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:03.324 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:03.324 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:03.325 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:03.325 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:03.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:03.325 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:03.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:03.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:03.326 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:03.326 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:03.327 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:03.327 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:03.328 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:03.328 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:03.328 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:03.329 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:03.329 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:03.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:03.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:03.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:03.330 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:03.330 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:03.330 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:03.331 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.713185_205.1", 2022-07-18T03:51:03.331 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:03.331 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:03.331 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:03.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.713185Z", 2022-07-18T03:51:03.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:03.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:03.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:03.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:03.333 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:03.333 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:03.333 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:03.334 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:03.334 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:03.334 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:03.334 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:03.335 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:03.335 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:03.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:03.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:03.336 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:03.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:03.336 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:03.336 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:03.337 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:03.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:03.337 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:03Z" 2022-07-18T03:51:03.337 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:03.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.713185_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.713185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:03Z" } 2022-07-18T03:51:03.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.713185_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.713185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:03.340 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:03.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.713185_205.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:03.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.713185_205.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:08.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:08.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:08.343 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:08.395 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:08.395 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:08.476 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:08.476 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:08.477 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:08.477 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:08.477 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:08.478 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:08.478 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:08.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:08.479 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:08.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:08.479 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:08.480 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:08.480 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:08.480 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:08.480 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:08.481 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:08.481 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:08.481 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:08.482 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:08.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:08.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:08.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:08.483 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:08.483 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:08.483 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:08.483 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:08.484 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:08.484 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:08.485 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:08.486 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:08.487 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:08.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:08.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:08.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:08.489 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:08.489 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:08.489 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:08.490 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:51:08.491 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:08.491 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:08.491 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:08.492 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:51:08.492 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:08.493 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:08.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:08.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:08.494 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:08.494 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:08.495 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:08.495 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:08.497 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:08.497 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:08.497 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:08.498 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:08.498 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:08.498 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:08.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:08.499 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:08.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:08.499 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:08.500 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:08.500 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:08.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:08.501 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:08Z" 2022-07-18T03:51:08.501 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:08.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:08Z" } 2022-07-18T03:51:08.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:08.504 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:08.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:08.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:13.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:13.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:13.509 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:13.561 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:13.562 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:13.625 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:13.625 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:13.625 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:13.626 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:13.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:13.628 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:13.629 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:13.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:13.631 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:13.632 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:13.633 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:13.633 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:13.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:13.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:13.634 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:13.634 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:13.634 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:51:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:13.635 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:51:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:13.636 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:13.637 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:13.638 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:13.638 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:13.638 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:13.639 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:13.639 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:13.639 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:13.639 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:13.640 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:13.640 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:13.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:13.640 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:13.641 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:13.641 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:13.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:13.642 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:13Z" 2022-07-18T03:51:13.642 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:13.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:13Z" } 2022-07-18T03:51:13.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:13.645 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:13.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:13.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:18.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:18.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:18.649 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:18.697 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:18.698 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:18.777 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:18.778 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:18.778 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:18.779 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:18.779 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:18.779 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:18.779 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:18.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:18.780 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:18.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:18.780 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:18.781 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:18.781 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:18.781 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:18.781 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:18.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:18.782 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:18.782 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:18.782 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:18.783 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:18.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:18.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:18.783 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:18.784 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:18.784 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:18.784 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:18.785 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:18.785 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:18.785 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:18.785 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:18.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:18.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:18.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:18.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:18.787 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:18.787 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:18.787 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:18.788 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.713185_205.1", 2022-07-18T03:51:18.788 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:18.788 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:18.788 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:18.789 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.713185Z", 2022-07-18T03:51:18.789 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:18.789 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:18.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:18.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:18.790 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:18.790 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:18.791 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:18.791 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:18.791 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:18.791 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:18.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:18.792 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:18.792 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:18.792 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:18.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:18.793 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:18.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:18.793 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:18.794 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:18.794 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:18.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:18.794 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:18Z" 2022-07-18T03:51:18.795 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:18.796 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.713185_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.713185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:18Z" } 2022-07-18T03:51:18.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.713185_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.713185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:18.797 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:18.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.713185_205.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:18.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.713185_205.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:23.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:23.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:23.801 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:23.855 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:23.855 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.707851_203.1", 2022-07-18T03:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.707851Z", 2022-07-18T03:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:23Z" 2022-07-18T03:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:23.952 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.707851_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.707851Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:23Z" } 2022-07-18T03:51:23.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.707851_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.707851Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:23.953 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:23.953 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.707851_203.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:23.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.707851_203.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:28.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:28.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:28.957 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:29.016 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:29.016 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:29.096 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116089.747425_207.1", 2022-07-18T03:51:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:09.747425Z", 2022-07-18T03:51:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:29.123 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:29.123 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:29.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:29.124 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:29.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:29.125 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:29.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:29.127 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:29Z" 2022-07-18T03:51:29.127 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:29.130 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116089.747425_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:09.747425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:29Z" } 2022-07-18T03:51:29.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116089.747425_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:09.747425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:29.132 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:29.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116089.747425_207.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:29.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116092.688198_260.1 target=1_1658116089.747425_207.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:34.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:34.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:34.138 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:34.194 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:34.195 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:34.274 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:34.275 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:34.275 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:34.275 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:34.276 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:34.276 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:34.276 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:34.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:34.277 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:34.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:34.277 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:34.277 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:34.278 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:34.278 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:34.278 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:34.278 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:34.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:34.279 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:34.279 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:34.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:34.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:34.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:34.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:34.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:34.281 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:34.282 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:34.282 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:34.283 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:34.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:34.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:34.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:34.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:34.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:34.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:34.284 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:34.285 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:34.285 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:34.285 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:51:34.285 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:34.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:34.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:51:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:34.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:34.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:34.289 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:34.289 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:34.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:34.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:34.290 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:34.290 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:34.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:34.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:34.291 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:34.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:34.292 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:34.292 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:34.292 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:34.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:34.293 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:34Z" 2022-07-18T03:51:34.293 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:34.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:34Z" } 2022-07-18T03:51:34.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:34.296 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:34.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:34.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:39.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:39.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:39.301 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:39.356 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:39.356 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:39.425 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:39.425 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:39.426 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:39.426 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:39.426 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:39.427 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:39.427 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:39.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:39.427 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:39.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:39.428 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:39.428 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:39.428 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:39.429 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:39.429 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:39.429 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:39.429 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:39.430 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:39.430 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:39.430 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:39.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:39.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:39.431 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:39.431 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:39.431 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:39.432 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:39.432 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:39.432 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:39.432 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:39.433 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:39.433 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:39.433 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:39.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:39.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:39.434 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:39.434 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:39.434 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:39.435 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:51:39.435 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:39.435 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:39.435 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:39.436 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:51:39.436 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:39.436 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:39.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:39.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:39.437 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:39.437 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:39.437 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:39.438 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:39.438 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:39.438 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:39.438 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:39.439 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:39.439 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:39.439 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:39.439 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:39.440 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:39.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:39.440 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:39.440 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:39.441 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:39.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:39.441 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:39Z" 2022-07-18T03:51:39.441 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:39.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:39Z" } 2022-07-18T03:51:39.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:39.444 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:39.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:39.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:44.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:44.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:44.448 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:44.505 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:44.506 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:44.589 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:44.590 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:44.590 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:44.590 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:44.591 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:44.591 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:44.591 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:44.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:44.592 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:44.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:44.593 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:44.593 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:44.593 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:44.594 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:44.594 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:44.594 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:44.595 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:44.595 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:44.595 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:44.595 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:44.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:44.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:44.596 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:44.596 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:44.597 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:44.597 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:44.597 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:44.597 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:44.598 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:44.598 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:44.598 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:44.599 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:44.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:44.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:44.599 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:44.600 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:44.600 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:44.600 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:51:44.600 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:44.601 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:44.601 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:44.601 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:51:44.601 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:44.602 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:44.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:44.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:44.602 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:44.603 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:44.603 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:44.603 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:44.603 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:44.604 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:44.604 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:44.604 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:44.604 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:44.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:44.607 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:44Z" 2022-07-18T03:51:44.607 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:44.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:44Z" } 2022-07-18T03:51:44.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:44.610 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:44.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:44.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:49.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:49.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:49.614 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:49.663 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:49.663 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:49.747 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:49.747 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:49.747 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:49.747 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:49.748 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:49.748 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:49.748 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:49.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:49.749 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:49.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:49.749 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:49.750 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:49.750 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:49.750 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:49.750 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:49.751 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:49.751 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:49.751 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:49.751 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:49.752 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:49.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:49.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:49.752 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:49.752 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:49.753 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:49.753 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:49.753 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:49.753 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:49.754 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:49.754 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:49.754 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:49.754 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:49.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:49.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:49.755 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:49.755 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:49.756 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:49.756 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:51:49.756 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:49.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:49.757 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:49.757 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:51:49.757 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:49.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:49.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:49.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:49.758 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:49.758 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:49.759 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:49.759 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:49.759 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:49.759 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:49.760 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:49.760 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:49.760 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:49.760 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:49.761 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:49.761 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:49.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:49.761 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:49.761 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:49.762 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:49.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:49.762 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:49Z" 2022-07-18T03:51:49.762 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:49.764 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:49Z" } 2022-07-18T03:51:49.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:49.766 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:49.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:49.767 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:54.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:54.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:54.771 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:51:54.827 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:51:54.828 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:51:54.906 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:51:54.907 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:51:54.907 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:51:54.908 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:51:54.908 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:51:54.909 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:51:54.909 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:54.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:54.910 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:51:54.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:54.912 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:51:54.912 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:54.913 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:54.913 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:51:54.914 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:54.915 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:51:54.915 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:54.915 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:51:54.916 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:54.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:54.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:54.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:54.918 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:51:54.918 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:54.919 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:54.919 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:51:54.920 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:54.921 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:54.921 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:54.921 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:51:54.922 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:54.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:54.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:54.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:54.924 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:51:54.924 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:54.925 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:54.925 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:51:54.926 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:54.926 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:54.927 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:54.927 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:51:54.928 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:54.928 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:54.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:54.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:51:54.930 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:51:54.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:51:54.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:51:54.932 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:51:54.932 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:51:54.933 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:51:54.933 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:51:54.934 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:51:54.934 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:51:54.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:54.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:51:54.936 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:51:54.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:54.938 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:51:54.938 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:51:54.938 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:51:54.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:51:54.939 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:51:54Z" 2022-07-18T03:51:54.939 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:51:54.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:51:54Z" } 2022-07-18T03:51:54.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:51:54.942 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:51:54.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:51:54.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:51:59.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:59.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:51:59.947 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:00.004 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:00.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:00.072 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:00.073 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:00.073 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:00.074 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:00.075 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:00.075 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:00.076 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:00.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:00.077 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:00.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:00.078 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:00.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:00.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:00.079 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:00.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:00.079 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:00.080 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:00.080 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:00.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:00.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:00.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:00.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:00.081 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:00.081 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:00.082 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:00.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:00.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:00.082 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:00.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:00.083 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:00.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:00.083 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:00.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:00.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:00.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:00.085 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:00.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:00.085 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:00.085 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:00.086 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:00.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:00.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:00.087 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:00.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:00.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:00.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:00.088 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:00.088 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:00.088 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:00.089 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:00.089 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:00.089 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:00.089 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:00.090 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:00.090 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:00.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:00.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:00.091 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:00.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:00.092 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:00.092 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:00.092 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:00.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:00.093 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:00Z" 2022-07-18T03:52:00.093 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:00.095 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:00Z" } 2022-07-18T03:52:00.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:00.096 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:00.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:00.097 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:05.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:05.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:05.100 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:05.154 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:05.155 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:05.229 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:05.229 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:05.230 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:05.230 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:05.230 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:05.231 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:05.231 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:05.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:05.231 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:05.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:05.232 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:05.232 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:05.232 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:05.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:05.233 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:05.233 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:05.233 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:05.234 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:05.234 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:05.234 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:05.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:05.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:05.235 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:05.235 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:05.236 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:05.236 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:05.236 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:05.236 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:05.237 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:05.237 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:05.237 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:05.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:05.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:05.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:05.238 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:05.238 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:05.239 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:05.239 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:05.239 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:05.239 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:05.240 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:05.240 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:05.240 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:05.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:05.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:05.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:05.241 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:05.241 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:05.242 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:05.242 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:05.242 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:05.242 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:05.243 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:05.243 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:05.243 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:05.243 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:05.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:05.244 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:05.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:05.244 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:05.245 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:05.245 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:05.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:05.245 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:05Z" 2022-07-18T03:52:05.246 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:05.247 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:05Z" } 2022-07-18T03:52:05.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:05.248 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:05.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:05.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:10.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:10.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:10.253 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:10.328 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:10.328 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:10.462 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:10.462 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:10.463 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:10.463 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:10.463 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:10.464 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:10.464 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:10.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:10.464 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:10.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:10.465 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:10.465 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:10.465 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:10.466 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:10.466 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:10.466 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:10.466 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:10.467 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:10.467 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:10.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:10.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:10.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:10.468 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:10.468 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:10.468 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:10.469 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:10.469 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:10.469 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:10.469 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:10.470 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:10.470 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:10.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:10.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:10.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:10.471 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:10.471 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:10.471 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:10.472 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:10.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:10.474 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:10.475 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:10.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:10.477 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:10.478 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:10Z" 2022-07-18T03:52:10.478 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:10.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:10Z" } 2022-07-18T03:52:10.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:10.481 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:10.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:10.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:15.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:15.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:15.486 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:15.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:15.542 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:15.631 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:15.633 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:15.633 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:15.634 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:15.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:15.636 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:15.637 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:15.638 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:15.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:15.640 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:15.641 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:15.642 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:15.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:15.644 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:15.645 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:15.646 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:15.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:15.648 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:15.649 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:15.650 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:15.651 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:15.651 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:15.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:15.652 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:15.652 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:15.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:15.652 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:15.653 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:15.653 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:15.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:15.654 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:15Z" 2022-07-18T03:52:15.654 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:15.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:15Z" } 2022-07-18T03:52:15.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:15.657 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:15.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:15.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:20.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:20.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:20.661 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:20.717 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:20.718 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:20.787 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:20.788 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:20.788 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:20.789 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:20.789 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:20.789 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:20.789 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:20.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:20.790 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:20.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:20.790 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:20.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:20.791 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:20.791 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:20.792 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:20.792 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:20.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:20.793 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:20.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:20.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:20.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:20.794 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:20.795 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:20.796 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:20.797 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:20.798 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:20.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:20.800 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:20.801 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:20.802 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:20.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:20.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:20.805 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:20Z" 2022-07-18T03:52:20.805 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:20.806 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:20Z" } 2022-07-18T03:52:20.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:20.808 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:20.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:20.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:25.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:25.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:25.812 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:25.869 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:25.869 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:25.947 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:25.948 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:25.948 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:25.948 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:25.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:25.950 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:25.951 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:25.952 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:25.952 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:25.956 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:25.957 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:25.958 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:25.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:25.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:25.961 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:25.962 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:25.962 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:25.964 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:25.964 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:25.965 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:25.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:25.967 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:25.968 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:25.968 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:25.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:25.969 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:25.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:25.969 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:25.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:25.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:25.971 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:25.972 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:25Z" 2022-07-18T03:52:25.973 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:25.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:25Z" } 2022-07-18T03:52:25.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:25.975 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:25.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:25.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:30.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:30.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:30.995 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:31.051 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:31.051 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:31.142 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:31.143 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:31.143 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:31.144 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:31.144 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:31.145 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:31.145 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:31.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:31.147 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:31.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:31.148 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:31.148 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:31.149 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:31.149 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:31.150 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:31.150 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:31.151 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:31.151 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:31.152 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:31.152 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:31.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:31.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:31.154 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:31.154 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:31.155 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:31.155 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:31.156 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:31.156 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:31.157 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:31.157 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:31.158 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:31.158 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:31.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:31.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:31.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:31.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:31.161 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:31.161 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:31.162 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:31.162 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:31.163 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:31.163 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:31.164 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:31.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:31.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:31.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:31.166 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:31.167 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:31.167 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:31.168 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:31.168 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:31.170 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:31.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:31.171 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:31.172 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:31.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:31.173 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:31.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:31.174 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:31Z" 2022-07-18T03:52:31.175 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:31.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:31Z" } 2022-07-18T03:52:31.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:31.179 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:31.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:31.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:36.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:36.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:36.182 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:36.260 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:36.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:36.372 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:36.372 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:36.372 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:36.373 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:36.373 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:36.373 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:36.374 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:36.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:36.374 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:36.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:36.375 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:36.375 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:36.376 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:36.376 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:36.376 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:36.376 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:36.377 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:36.377 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:36.377 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:36.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:36.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:36.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:36.379 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:36.379 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:36.379 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:36.380 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:36.380 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:36.381 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:36.381 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:36.381 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:36.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:36.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:36.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:36.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:36.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:36.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:36.383 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:36.383 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:36.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:36.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:36.384 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:36.384 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:36.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:36.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:36.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:36.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:36.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:36.386 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:36.386 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:36.387 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:36.387 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:36.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:36.387 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:36.388 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:36.388 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:36.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:36.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:36.389 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:36.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:36.390 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:36.391 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:36.391 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:36.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:36.392 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:36Z" 2022-07-18T03:52:36.393 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:36.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:36Z" } 2022-07-18T03:52:36.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:36.397 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:36.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:36.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:41.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:41.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:41.402 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:41.451 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:41.452 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:41Z" 2022-07-18T03:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:41.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:41Z" } 2022-07-18T03:52:41.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:41.546 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:41.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:41.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:46.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:46.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:46.550 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:46.606 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:46.607 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:46.676 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:46Z" 2022-07-18T03:52:46.694 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:46.695 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:46Z" } 2022-07-18T03:52:46.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:46.697 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:46.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:46.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:51.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:51.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:51.700 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:51.756 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:51.757 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:51.849 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:51.849 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:51.849 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:51.850 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:51.850 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:51.851 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:51.851 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:51.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:51.851 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:51.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:51.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:51.852 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:51.852 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:51.853 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:51.853 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:51.853 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:51.853 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:51.854 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:51.854 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:51.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:51.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:51.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:51.855 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:51.856 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:51.856 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:51.856 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:51.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:51.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:51.857 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:51.857 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:51.858 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:51.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:51.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:51.859 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:51.860 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:51.860 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:51.860 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:51.861 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:51.861 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:51.861 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:51.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:51.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:51.862 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:51.863 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:51.863 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:51.863 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:51.863 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:51.864 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:51.864 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:51.865 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:51.865 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:51.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:51.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:51.867 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:51.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:51.868 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:51.868 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:51.869 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:51.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:51.869 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:51Z" 2022-07-18T03:52:51.870 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:51.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:51Z" } 2022-07-18T03:52:51.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:51.874 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:51.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:51.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:52:56.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:56.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:52:56.878 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:52:56.931 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:52:56.933 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:52:57.005 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:52:57.006 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:52:57.006 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:52:57.006 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:52:57.007 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:52:57.007 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:52:57.007 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:57.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:57.008 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:52:57.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:57.011 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:52:57.011 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:57.011 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:57.012 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:52:57.012 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:57.013 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:52:57.014 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:57.014 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:52:57.015 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:57.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:57.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:57.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:57.018 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:52:57.019 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:57.019 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:57.020 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:52:57.020 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:57.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:57.021 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:57.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:52:57.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:57.024 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:57.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:57.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:57.027 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:52:57.028 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:57.030 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:52:57.031 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:57.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:57.032 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:57.033 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:52:57.033 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:57.034 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:57.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:57.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:52:57.035 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:52:57.036 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:52:57.036 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:52:57.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:52:57.037 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:52:57.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:52:57.039 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:52:57.040 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:52:57.041 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:52:57.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:57.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:52:57.043 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:52:57.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:57.044 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:52:57.046 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:52:57.047 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:52:57.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:52:57.051 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:52:56Z" 2022-07-18T03:52:57.051 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:52:57.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:52:56Z" } 2022-07-18T03:52:57.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:52:57.054 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:52:57.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:52:57.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:02.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:02.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:02.059 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:02.114 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:02.115 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:02.191 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:02.191 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:02.191 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:02.192 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:02.192 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:02.192 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:02.192 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:02.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:02.193 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:02.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:02.193 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:02.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:02.194 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:02.194 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:02.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:02.195 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:02.195 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:02.195 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:02.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:02.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:02.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:02.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:02.197 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:02.198 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:02.198 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:02.198 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:02.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:02.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:02.199 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:02.199 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:02.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:02.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:02.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:02.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:02.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:02.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:02.202 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:02.202 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:53:02.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:02.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:02.203 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:02.203 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:53:02.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:02.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:02.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:02.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:02.205 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:02.205 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:02.206 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:02.206 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:02.206 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:02.207 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:02.207 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:02.207 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:02.208 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:02.208 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:02.208 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:02Z" 2022-07-18T03:53:02.211 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:02.212 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:02Z" } 2022-07-18T03:53:02.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:02.213 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:02.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:02.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:07.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:07.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:07.220 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:07.275 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:07.276 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:07.361 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:07.362 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:07.362 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:07.362 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:07.362 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:07.363 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:07.363 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:07.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:07.363 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:07.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:07.364 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:07.365 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:07.365 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:07.365 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:07.365 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:07.366 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:07.366 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:07.366 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:07.367 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:07.367 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:07.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:07.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:07.368 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:07.368 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:07.368 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:07.368 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:07.369 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:07.369 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:07.369 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:07.369 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:07.370 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:07.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:07.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:07.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:07.371 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:07.371 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:07.371 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:07.371 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:53:07.372 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:07.372 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:07.372 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:07.372 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:53:07.373 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:07.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:07.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:07.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:07.374 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:07.374 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:07.374 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:07.375 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:07.375 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:07.375 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:07.375 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:07.375 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:07.376 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:07.376 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:07.376 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:07.376 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:07.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:07.377 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:07.377 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:07.377 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:07.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:07.378 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:07Z" 2022-07-18T03:53:07.378 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:07.380 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:07Z" } 2022-07-18T03:53:07.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:07.381 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:07.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:07.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:12.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:12.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:12.385 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:12.441 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:12.442 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:12.506 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:12.515 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:12.515 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:12.515 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:12.516 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:12.516 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:12.516 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:12.516 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:12.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:12.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:12.517 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:12.517 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:12.518 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:12.518 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:12.518 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:12.518 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:12.519 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:12.519 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:12.519 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:12.520 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:12.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:12.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:12.520 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:12.521 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:12.521 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:12.521 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:53:12.521 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:12.522 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:12.522 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:12.522 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:53:12.522 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:12.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:12.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:12.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:12.524 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:12.524 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:12.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:12.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:12.525 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:12.525 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:12.525 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:12.525 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:12.526 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:12.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:12.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:12.526 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:12.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:12.527 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:12.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:12.528 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:12.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:12.528 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:12Z" 2022-07-18T03:53:12.529 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:12.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:12Z" } 2022-07-18T03:53:12.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:12.531 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:12.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:12.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:17.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:17.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:17.536 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:17.593 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:17.594 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:17.685 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116092.688198_260.1", 2022-07-18T03:53:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:48:12.688198Z", 2022-07-18T03:53:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:17Z" 2022-07-18T03:53:17.705 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:17.707 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116092.688198_260.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:48:12.688198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:17Z" } 2022-07-18T03:53:17.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116092.688198_260.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:48:12.688198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:17.708 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:17.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116092.688198_260.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:17.709 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:22.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2022-07-18T03:53:22.721 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-39 2022-07-18T03:53:22.723 DEBUG:boto:path=/mlvank-39/ 2022-07-18T03:53:22.724 DEBUG:boto:auth_path=/mlvank-39/ 2022-07-18T03:53:22.725 DEBUG:boto:Method: PUT 2022-07-18T03:53:22.726 DEBUG:boto:Path: /mlvank-39/ 2022-07-18T03:53:22.727 DEBUG:boto:Data: 2022-07-18T03:53:22.728 DEBUG:boto:Headers: {} 2022-07-18T03:53:22.729 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:53:22.730 DEBUG:boto:Port: 8000 2022-07-18T03:53:22.731 DEBUG:boto:Params: {} 2022-07-18T03:53:22.732 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:53:22.734 DEBUG:boto:Token: None 2022-07-18T03:53:22.735 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:53:22 GMT /mlvank-39/ 2022-07-18T03:53:22.736 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:BKc9QYq31W1m9bVrv7Ig/DJiOTM= 2022-07-18T03:53:22.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:53:22 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:BKc9QYq31W1m9bVrv7Ig/DJiOTM=', 'Content-Length': '0'} 2022-07-18T03:53:22.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000838f1c2242c5da50-0062d4d932-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:53:22.760 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-40 2022-07-18T03:53:22.760 DEBUG:boto:path=/mlvank-40/ 2022-07-18T03:53:22.761 DEBUG:boto:auth_path=/mlvank-40/ 2022-07-18T03:53:22.761 DEBUG:boto:Method: PUT 2022-07-18T03:53:22.762 DEBUG:boto:Path: /mlvank-40/ 2022-07-18T03:53:22.762 DEBUG:boto:Data: 2022-07-18T03:53:22.763 DEBUG:boto:Headers: {} 2022-07-18T03:53:22.764 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:53:22.765 DEBUG:boto:Port: 8000 2022-07-18T03:53:22.766 DEBUG:boto:Params: {} 2022-07-18T03:53:22.767 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:53:22.768 DEBUG:boto:Token: None 2022-07-18T03:53:22.769 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:53:22 GMT /mlvank-40/ 2022-07-18T03:53:22.771 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:vUdbcc1hTIbHUMm/kRVsaV9rqSo= 2022-07-18T03:53:22.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:53:22 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:vUdbcc1hTIbHUMm/kRVsaV9rqSo=', 'Content-Length': '0'} 2022-07-18T03:53:22.810 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc0c517829440b01-0062d4d932-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:53:22.811 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-41 2022-07-18T03:53:22.812 DEBUG:boto:path=/mlvank-41/ 2022-07-18T03:53:22.813 DEBUG:boto:auth_path=/mlvank-41/ 2022-07-18T03:53:22.814 DEBUG:boto:Method: PUT 2022-07-18T03:53:22.815 DEBUG:boto:Path: /mlvank-41/ 2022-07-18T03:53:22.816 DEBUG:boto:Data: 2022-07-18T03:53:22.817 DEBUG:boto:Headers: {} 2022-07-18T03:53:22.818 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:53:22.819 DEBUG:boto:Port: 8001 2022-07-18T03:53:22.820 DEBUG:boto:Params: {} 2022-07-18T03:53:22.821 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:53:22.822 DEBUG:boto:Token: None 2022-07-18T03:53:22.823 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:53:22 GMT /mlvank-41/ 2022-07-18T03:53:22.824 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:nB6QESk48jXu7+syMvmcQGh3P24= 2022-07-18T03:53:22.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:53:22 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:nB6QESk48jXu7+syMvmcQGh3P24=', 'Content-Length': '0'} 2022-07-18T03:53:22.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f3a8900fb077ef8-0062d4d932-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:53:22.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:22.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:22.874 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:22.921 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:22.922 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:23.031 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:53:23.108 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:53:23.109 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:53:23.110 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:23.110 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:53:23.111 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:53:23.111 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:53:23.112 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:53:23.112 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:53:23.113 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:53:23.113 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:53:23.114 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:23.114 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:53:22.849523Z" 2022-07-18T03:53:23.115 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:53:23.115 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:53:23.116 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:53:23.116 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:53:23.116 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:53:23.117 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:53:23.119 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:23.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:53:23.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:23.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:23.123 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:23.172 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:23.173 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:23.240 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:23.240 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:23.241 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:23.241 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:23.241 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:23.242 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:23.242 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:23.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:23.243 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:23.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:23.243 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:23.243 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:23.244 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:23.244 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:23.244 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:23.244 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:23.245 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:23.245 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:23.245 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:23.246 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:23.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:23.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:23.246 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:23.247 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:23.247 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:23.247 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:23.247 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:23.248 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:23.248 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:23.248 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:23.249 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:23.249 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:23.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:23.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:23.250 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:23.250 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:23.250 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:23.250 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.749815_261.1", 2022-07-18T03:53:23.251 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:23.251 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:23.251 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:23.251 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.749815Z", 2022-07-18T03:53:23.252 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:23.252 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:23.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:23.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:23.253 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:23.253 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:23.253 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:23.254 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:23.254 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:23.254 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:23.255 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:23.255 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:23.255 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:23.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:23.256 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:23.256 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:23.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:23.257 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:23.257 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:23.258 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:23.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:23.258 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:23Z" 2022-07-18T03:53:23.258 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:23.260 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.749815_261.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.749815Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:23Z" } 2022-07-18T03:53:23.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.749815_261.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.749815Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:23.261 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:23.262 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.749815_261.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:23.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116402.849523_272.1 target=1_1658116402.749815_261.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:28.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:28.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:28.266 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:28.321 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:28.322 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:28.396 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:28.397 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:28.398 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:28.398 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:28.399 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:28.399 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:28.400 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:28.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:28.401 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:28.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:28.402 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:28.402 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:28.403 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:28.403 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:28.404 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:28.405 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:28.405 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:28.406 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:28.406 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:28.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:28.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:28.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:28.408 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:28.409 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:28.409 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:28.410 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:28.410 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:28.411 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:28.411 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:28.411 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:28.412 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:28.412 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:28.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:28.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:28.414 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:28.414 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:28.415 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:28.415 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:28.416 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:28.416 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:28.417 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:28.417 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:28.417 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:28.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:28.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:28.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:28.419 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:28.419 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:28.420 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:28.420 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:28.421 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:28.421 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:28.422 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:28.422 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:28.423 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:28.423 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:28.424 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:28.424 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:28.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:28.425 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:28.425 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:28.425 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:28.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:28.426 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:28Z" 2022-07-18T03:53:28.427 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:28.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:28Z" } 2022-07-18T03:53:28.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:28.431 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:28.432 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:28.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:33.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:33.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:33.436 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:33.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:33.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:33.554 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:33Z" 2022-07-18T03:53:33.583 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:33.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:33Z" } 2022-07-18T03:53:33.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:33.588 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:33.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:33.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:38.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:38.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:38.593 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:38.649 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:38.650 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:38.784 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:38.785 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:38.786 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:38.786 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:38.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:38.787 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:38.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:38.788 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:38.789 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:38.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:38.791 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:38.792 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:38.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:38.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:38.794 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:38.794 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:38.794 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:38.795 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:38.795 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:38.795 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:38.795 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:38.796 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:38.796 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:38.796 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:38.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:38.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:38.797 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:38.797 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:38.798 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:38.798 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:38.798 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:38.798 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:38.799 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:38.799 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:38.799 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:38.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:38.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:38.800 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:38.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:38.800 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:38.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:38.801 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:38.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:38.801 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:38Z" 2022-07-18T03:53:38.802 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:38.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:38Z" } 2022-07-18T03:53:38.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:38.804 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:38.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:38.805 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:43.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:43.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:43.809 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:43.874 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:43.874 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:43.976 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:43.976 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:43.977 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:43.978 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:43.978 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:43.979 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:43.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:43.980 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:43.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:43.981 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:43.982 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:43.982 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:43.983 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:43.983 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:43.984 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:43.984 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:43.985 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:43.985 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:43.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:43.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:43.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:43.988 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:43.988 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:43.989 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:43.989 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:43.990 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:43.990 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:43.991 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:43.991 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:43.992 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:43.992 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:43.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:43.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:43.994 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:43.994 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:43.995 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:43.995 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:43.996 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:43.997 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:43.997 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:43.998 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:43.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:43.999 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:44.000 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:44.001 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:44.002 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:44.003 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:44.004 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:43Z" 2022-07-18T03:53:44.004 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:44.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:43Z" } 2022-07-18T03:53:44.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:44.007 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:44.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:44.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:49.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:49.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:49.012 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:49.070 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:49.071 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:49.134 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:49Z" 2022-07-18T03:53:49.152 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:49.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:49Z" } 2022-07-18T03:53:49.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:49.157 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:49.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:49.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:54.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:54.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:54.163 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:54.215 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:54.215 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:54.273 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:54.274 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:54.274 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:54.274 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:54.274 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:54.275 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:54.275 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:54.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:54.275 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:54.276 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:54.277 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:54.279 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:54.280 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:54.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:54.282 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:54.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:54.285 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:54.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:54.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:54.288 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:54.289 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:54Z" 2022-07-18T03:53:54.289 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:54.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:54Z" } 2022-07-18T03:53:54.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:54.292 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:54.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:54.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:53:59.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:59.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:53:59.297 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:53:59.345 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:53:59.346 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:53:59.423 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:53:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:53:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:53:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:53:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:53:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:53:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:59.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:53:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:53:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:53:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:53:59.428 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:59.428 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:53:59.428 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:59.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:59.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:59.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:59.429 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:53:59.430 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:59.430 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:59.430 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:53:59.430 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:59.431 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:59.431 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:59.431 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:53:59.431 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:59.432 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:59.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:59.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:59.432 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:53:59.433 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:59.433 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:59.433 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:53:59.433 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:59.434 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:59.434 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:59.434 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:53:59.434 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:59.435 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:59.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:59.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:53:59.435 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:53:59.436 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:53:59.436 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:53:59.436 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:53:59.436 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:53:59.437 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:53:59.437 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:53:59.437 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:53:59.437 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:53:59.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:59.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:53:59.438 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:53:59.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:59.439 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:53:59.439 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:53:59.439 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:53:59.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:53:59.440 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:53:59Z" 2022-07-18T03:53:59.440 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:53:59.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:53:59Z" } 2022-07-18T03:53:59.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:53:59.443 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:53:59.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:53:59.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:04.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:04.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:04.446 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:04.501 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:04.502 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:04.565 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:04.565 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:04.565 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:04.566 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:04.566 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:04.566 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:04.566 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:04.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:04.568 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:04.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:04.568 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:04.569 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:04.570 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:04.571 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:04.572 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:04.573 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:04.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:04.575 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:04.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:04.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:04.578 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:04.578 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:04.579 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:04.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:04Z" 2022-07-18T03:54:04.583 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:04.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:04Z" } 2022-07-18T03:54:04.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:04.585 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:04.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:04.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:09.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:09.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:09.590 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:09.661 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:09.661 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:09.752 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:09.752 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:09.753 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:09.753 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:09.753 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:09.754 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:09.754 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:09.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:09.754 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:09.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:09.755 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:09.755 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:09.756 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:09.756 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:09.756 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:09.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:09.757 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:09.757 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:09.757 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:09.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:09.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:09.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:09.758 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:09.759 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:09.759 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:09.760 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:09.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:09.762 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:09.763 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:09.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:09.765 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:09.766 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:09.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:09.768 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:09.769 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:09.770 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:09Z" 2022-07-18T03:54:09.770 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:09.772 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:09Z" } 2022-07-18T03:54:09.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:09.773 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:09.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:09.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:14.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:14.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:14.777 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:14.827 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:14.828 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:14.905 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:14.905 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:14.905 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:14.905 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:14.906 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:14.906 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:14.906 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:14.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:14.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:14.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:14.907 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:14.908 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:14.908 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:14.908 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:14.908 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:14.909 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:14.909 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:14.909 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:14.909 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:14.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:14.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:14.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:14.910 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:14.911 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:14.911 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:14.911 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:14.911 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:14.912 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:14.912 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:14.912 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:14.912 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:14.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:14.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:14.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:14.914 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:14.914 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:14.914 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:14.914 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:14.915 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:14.915 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:14.915 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:14.915 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:14.916 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:14.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:14.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:14.916 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:14.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:14.917 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:14.917 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:14.917 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:14.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:14.918 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:14Z" 2022-07-18T03:54:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:14.919 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:14Z" } 2022-07-18T03:54:14.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:14.921 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:14.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:14.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:19.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:19.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:19.925 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:19.979 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:19.980 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:20.051 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:20.052 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:20.053 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:20.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:20.054 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:20.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:20.055 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:20.055 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:20.055 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:20.055 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:20.056 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:20.056 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:20.056 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:20.057 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:20.057 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:20.057 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:20.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:20.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:20.058 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:20.059 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:20.059 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:20.060 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:20.060 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:20.060 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:20.061 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:20.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:20.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:20.063 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:20.063 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:20.064 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:20.064 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:20.065 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:20.065 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:20.066 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:20.066 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:20.067 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:20.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:20.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:20.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:20.069 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:20.069 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:20.070 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:20.070 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:20.071 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:20.071 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:20.072 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:20.072 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:20.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:20.073 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:20.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:20.075 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:20.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:20.076 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:20.076 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:20.077 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:20.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:20.078 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:20Z" 2022-07-18T03:54:20.078 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:20.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:20Z" } 2022-07-18T03:54:20.080 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:20.081 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:20.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:20.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:25.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:25.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:25.085 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:25.140 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:25.141 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:25.214 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:25.214 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:25.215 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:25.215 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:25.215 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:25.215 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:25.216 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:25.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:25.216 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:25.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:25.217 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:25.217 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:25.218 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:25.218 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:25.218 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:25.218 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:25.219 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:25.219 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:25.219 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:25.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:25.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:25.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:25.220 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:25.222 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:25.222 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:25.222 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:25.223 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:25.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:25.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:25.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:25.224 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:25.224 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:25.224 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:25.225 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:25.225 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:25.225 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:25.225 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:25.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:25.227 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:25.227 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:25.228 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:25.228 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:25.228 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:25.228 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:25.229 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:25.229 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:25.229 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:25.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:25.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:25.230 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:25.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:25.231 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:25.231 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:25.231 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:25.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:25.232 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:25Z" 2022-07-18T03:54:25.232 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:25.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:25Z" } 2022-07-18T03:54:25.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:25.235 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:25.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:25.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:30.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:30.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:30.238 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:30.294 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:30.294 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:30.386 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:30.387 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:30.388 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:30.388 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:30.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:30.388 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:30.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:30.389 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:30.389 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:30.390 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:30.390 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:30.390 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:30.390 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:30.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:30.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:30.393 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:30.393 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:30.393 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:30.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:30.395 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:30.396 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:30.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:30.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:30.399 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:30.401 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:30.402 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:30.403 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:30Z" 2022-07-18T03:54:30.403 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:30.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:30Z" } 2022-07-18T03:54:30.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:30.406 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:30.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:30.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:35.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:35.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:35.410 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:35.459 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:35.459 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:35.520 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:35.521 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:35.521 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:35.521 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:35.522 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:35.522 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:35.522 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:35.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:35.523 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:35.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:35.523 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:35.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:35.524 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:35.527 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:35.528 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:35.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:35.530 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:35.531 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:35.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:35.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:35.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:35.537 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:35Z" 2022-07-18T03:54:35.537 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:35.538 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:35Z" } 2022-07-18T03:54:35.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:35.540 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:35.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:35.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:40.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:40.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:40.544 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:40.599 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:40.599 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:40.693 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:40.694 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:40.695 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:40.695 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:40.696 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:40.696 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:40.697 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:40.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:40.698 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:40.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:40.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:40.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:40.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:40.717 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:40.717 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:40.718 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:40.718 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:40.719 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:40.719 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:40.720 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:40.720 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:40.721 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:40.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:40.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:40.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:40.723 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:40.723 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:40.724 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:40.724 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:40.725 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:40.725 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:40.726 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:40.726 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:40.727 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:40.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:40.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:40.729 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:40.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:40.730 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:40.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:40.731 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:40.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:40.732 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:40Z" 2022-07-18T03:54:40.733 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:40.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:40Z" } 2022-07-18T03:54:40.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:40.737 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:40.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:40.739 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:45.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:45.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:45.742 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:45.955 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:45.963 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:45.963 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:45.963 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:45.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:45.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:45.964 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:45.964 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:45.964 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:45.965 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:45.965 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:45.965 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:45.965 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:45.966 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:45.966 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:45.966 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:45.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:45.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:45.967 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:45.968 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:45.968 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:45.968 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:45.968 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:45.969 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:45.969 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:45.969 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:45.969 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:45.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:45.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:45.970 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:45.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:45.970 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:45.971 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:45.971 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:45.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:45.971 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:45Z" 2022-07-18T03:54:45.972 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:45.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:45Z" } 2022-07-18T03:54:45.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:45.974 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:45.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:45.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:50.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:50.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:50.978 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:51.035 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:51.035 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:51.120 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:51.121 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:51.121 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:51.122 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:51.122 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:51.122 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:51.122 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:51.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:51.123 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:51.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:51.126 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:51.126 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:51.126 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:51.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:51.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:51.127 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:51.128 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:51.128 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:51.128 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:51.128 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:51.129 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:51.129 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:51.129 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:51.130 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:51.130 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:51.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:51.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:51.131 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:51.131 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:51.131 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:51.132 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:51.132 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:51.132 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:51.133 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:51.133 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:51.133 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:51.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:51.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:51.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:51.134 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:51.134 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:51.135 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:51.135 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:51.135 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:51.135 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:51.136 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:51.136 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:51.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:51.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:51.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:51.137 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:51.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:51.137 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:51.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:51.138 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:51.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:51.138 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:51Z" 2022-07-18T03:54:51.139 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:51.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:51Z" } 2022-07-18T03:54:51.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:51.141 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:51.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:51.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:54:56.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:56.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:54:56.145 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:54:56.206 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:54:56.207 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:54:56.368 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:54:56.368 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:54:56.369 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:54:56.369 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:54:56.369 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:54:56.370 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:54:56.370 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:56.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:56.371 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:54:56.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:56.371 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:54:56.372 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:56.372 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:56.372 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:54:56.372 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:56.373 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:54:56.373 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:56.373 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:54:56.373 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:56.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:56.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:56.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:56.374 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:54:56.375 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:56.375 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:56.375 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:54:56.375 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:56.376 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:56.376 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:56.377 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:54:56.377 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:56.377 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:56.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:56.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:56.378 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:54:56.379 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:56.379 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:56.379 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:54:56.379 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:56.380 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:56.380 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:56.380 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:54:56.380 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:56.381 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:56.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:56.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:54:56.382 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:54:56.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:54:56.383 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:54:56.383 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:54:56.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:54:56.384 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:54:56.384 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:54:56.384 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:54:56.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:54:56.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:56.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:54:56.385 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:54:56.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:56.386 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:54:56.386 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:54:56.386 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:54:56.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:54:56.387 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:54:56Z" 2022-07-18T03:54:56.387 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:54:56.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:54:56Z" } 2022-07-18T03:54:56.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:54:56.390 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:54:56.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:54:56.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:01.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:01.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:01.394 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:01.449 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:01.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:01.529 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:01.530 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:01.530 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:01.530 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:01.530 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:01.531 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:01.531 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:01.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:01.532 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:01.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:01.532 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:01.532 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:01.533 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:01.533 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:01.533 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:01.533 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:01.534 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:01.534 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:01.534 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:01.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:01.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:01.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:01.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:01.536 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:01.536 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:01.536 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:01.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:01.537 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:01.537 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:01.537 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:01.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:01.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:01.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:01.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:01.538 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:01.539 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:01.539 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:01.539 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:01.540 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:01.540 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:01.540 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:01.540 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:01.541 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:01.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:01.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:01.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:01.542 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:01.542 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:01.542 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:01.542 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:01.543 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:01.543 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:01.543 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:01.544 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:01.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:01.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:01.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:01.545 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:01.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:01.545 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:01.545 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:01.546 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:01.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:01.546 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:01Z" 2022-07-18T03:55:01.546 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:01.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:01Z" } 2022-07-18T03:55:01.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:01.549 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:01.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:01.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:06.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:06.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:06.554 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:06.610 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:06.610 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:06.687 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:06.687 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:06.688 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:06.688 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:06.688 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:06.689 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:06.689 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:06.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:06.689 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:06.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:06.690 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:06.690 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:06.690 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:06.691 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:06.691 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:06.691 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:06.692 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:06.692 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:06.692 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:06.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:06.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:06.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:06.693 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:06.693 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:06.694 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:06.694 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:06.694 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:06.694 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:06.695 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:06.695 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:06.695 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:06.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:06.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:06.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:06.696 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:06.697 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:06.697 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:06.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:06.697 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:06.698 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:06.698 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:06.698 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:06.698 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:06.699 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:06.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:06.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:06.700 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:06.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:06.700 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:06.700 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:06.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:06.701 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:06.701 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:06.701 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:06.702 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:06.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:06.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:06.702 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:06.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:06.703 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:06.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:06.703 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:06.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:06.704 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:06Z" 2022-07-18T03:55:06.704 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:06.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:06Z" } 2022-07-18T03:55:06.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:06.707 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:06.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:06.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:11.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:11.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:11.711 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:11.764 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:11.765 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:11.839 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:11.840 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:11.841 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:11.841 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:11.842 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:11.842 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:11.843 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:11.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:11.843 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:11.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:11.844 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:11.845 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:11.845 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:11.845 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:11.845 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:11.846 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:11.846 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:11.846 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:11.847 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:11.847 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:11.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:11.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:11.848 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:11.848 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:11.848 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:11.848 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:11.849 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:11.849 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:11.850 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:11.850 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:11.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:11.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:11.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:11.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:11.853 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:11.853 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:11.853 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:11.853 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:11.854 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:11.854 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:11.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:11.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:11.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:11.857 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:11.857 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:11.858 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:11.858 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:11.859 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:11.859 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:11.860 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:11.860 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:11.861 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:11.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:11.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:11.863 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:11.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:11.864 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:11.864 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:11.865 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:11.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:11.866 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:11Z" 2022-07-18T03:55:11.866 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:11.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:11Z" } 2022-07-18T03:55:11.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:11.870 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:11.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:11.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:16.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:16.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:16.875 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:16.931 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:16.931 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:17.016 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:16Z" 2022-07-18T03:55:17.033 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:17.034 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:16Z" } 2022-07-18T03:55:17.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:17.035 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:17.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:17.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:22.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:22.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:22.039 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:22.095 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:22.095 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:22.177 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:22.178 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:22.178 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:22.178 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:22.179 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:22.179 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:22.179 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:22.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:22.180 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:22.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:22.180 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:22.181 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:22.181 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:22.181 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:22.181 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:22.182 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:22.182 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:22.182 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:22.183 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:22.183 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:22.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:22.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:22.185 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:22.185 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:22.185 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:22.186 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:22.187 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:22.187 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:22.188 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:22.188 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:22.189 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:22.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:22.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:22.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:22.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:22.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:22.192 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:22.192 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:22.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:22.193 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:22.193 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:22.194 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:22.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:22.195 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:22.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:22.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:22.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:22.196 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:22.196 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:22.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:22.196 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:22.197 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:22.197 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:22.197 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:22.198 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:22.198 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:22.198 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:22.198 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:22.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:22.199 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:22.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:22.199 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:22.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:22.200 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:22Z" 2022-07-18T03:55:22.200 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:22.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:22Z" } 2022-07-18T03:55:22.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:22.203 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:22.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:22.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:27.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:27.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:27.207 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:27.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:27.261 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:27.351 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:27Z" 2022-07-18T03:55:27.367 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:27.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:27Z" } 2022-07-18T03:55:27.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:27.370 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:27.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:27.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:32.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:32.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:32.374 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:32.431 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:32.431 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:32.500 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:32.500 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:32.501 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:32.501 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:32.502 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:32.502 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:32.502 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:32.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:32.503 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:32.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:32.504 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:32.505 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:32.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:32.505 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:32.506 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:32.506 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:32.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:32.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:32.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:32.507 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:32.507 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:32.507 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:32.508 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:32.508 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:32.508 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:32.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:32.510 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:32.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:32.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:32.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:32.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:32.514 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:32.514 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:32.514 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:32.514 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:32.515 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:32.515 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:32.515 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:32.516 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:32.516 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:32.516 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:32.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:32.517 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:32.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:32.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:32.519 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:32Z" 2022-07-18T03:55:32.519 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:32.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:32Z" } 2022-07-18T03:55:32.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:32.522 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:32.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:32.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:37.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:37.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:37.526 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:37.582 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:37.582 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:37.660 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:37.661 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:37.661 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:37.661 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:37.662 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:37.662 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:37.662 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:37.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:37.663 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:37.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:37.663 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:37.663 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:37.664 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:37.664 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:37.664 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:37.664 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:37.665 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:37.665 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:37.665 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:37.665 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:37.666 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:37.667 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:37.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:37.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:37.669 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:37.669 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:37.669 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:37.670 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:37.670 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:37.670 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:37.670 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:37.671 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:37.671 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:37.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:37.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:37.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:37.672 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:37.672 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:37.673 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:37.673 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:37.673 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:37.673 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:37.673 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:37.674 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:37.674 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:37.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:37.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:37.675 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:37.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:37.675 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:37.675 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:37Z" 2022-07-18T03:55:37.676 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:37.678 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:37Z" } 2022-07-18T03:55:37.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:37.679 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:37.679 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:37.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:42.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:42.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:42.683 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:42.739 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:42.739 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:42.821 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:42.835 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:42.835 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:42.835 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:42.835 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:42.839 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:42Z" 2022-07-18T03:55:42.839 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:42.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:42Z" } 2022-07-18T03:55:42.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:42.841 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:42.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:42.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:47.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:47.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:47.846 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:47.903 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:47.903 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:47.968 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:47.968 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:47.968 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:47.969 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:47.969 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:47.969 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:47.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:47.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:47.970 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:47.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:47.971 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:47.971 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:47.971 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:47.972 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:47.972 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:47.972 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:47.972 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:47.973 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:47.973 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:47.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:47.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:47.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:47.974 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:47.974 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:47.974 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:47.975 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:47.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:47.977 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:47.978 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:47.978 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:47.978 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:47.979 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:47.979 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:47.979 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:47.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:47.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:47.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:47.980 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:47.980 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:47.981 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:47.982 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:47.983 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:47.983 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:47.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:47.983 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:47.984 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:47.984 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:47.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:47.984 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:47Z" 2022-07-18T03:55:47.985 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:47.986 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:47Z" } 2022-07-18T03:55:47.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:47.987 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:47.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:47.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:52.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:52.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:52.991 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:53.045 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:53.045 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:53.124 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:53.125 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:53.125 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:53.125 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:53.125 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:53.126 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:53.126 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:53.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:53.127 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:53.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:53.127 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:53.127 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:53.128 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:53.128 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:53.128 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:53.128 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:53.129 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:53.129 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:53.129 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:53.130 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:53.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:53.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:53.131 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:53.131 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:53.131 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:53.132 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:53.132 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:53.132 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:53.132 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:53.133 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:53.133 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:53.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:53.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:53.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:53.134 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:53.134 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:53.135 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:53.135 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:53.135 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:53.135 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:53.136 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:53.136 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:53.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:53.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:53.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:53.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:53.137 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:53.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:53.138 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:53.138 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:53.138 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:53.139 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:53.139 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:53.139 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:53.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:53.140 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:53.140 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:53.140 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:53.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:53.141 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:53.141 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:53.141 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:53.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:53.142 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:53Z" 2022-07-18T03:55:53.142 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:53.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:53Z" } 2022-07-18T03:55:53.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:53.145 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:53.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:53.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:55:58.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:58.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:55:58.149 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:55:58.206 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:55:58.207 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:55:58.276 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:55:58.276 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:55:58.277 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:55:58.277 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:55:58.277 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:55:58.277 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:55:58.278 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:58.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:58.278 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:55:58.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:58.279 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:55:58.279 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:58.279 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:58.279 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:55:58.280 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:58.280 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:55:58.280 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:58.280 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:55:58.281 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:58.281 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:58.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:58.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:58.282 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:55:58.282 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:58.282 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:58.283 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:55:58.283 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:58.283 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:58.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:58.284 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:55:58.284 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:58.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:58.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:58.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:58.285 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:55:58.285 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:58.285 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:58.286 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:55:58.286 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:58.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:58.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:58.287 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:55:58.287 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:58.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:58.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:58.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:55:58.288 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:55:58.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:55:58.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:55:58.289 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:55:58.289 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:55:58.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:55:58.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:55:58.290 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:55:58.290 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:55:58.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:55:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:55:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:55:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:55:58.292 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:55:58.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:55:58.292 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:55:58Z" 2022-07-18T03:55:58.292 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:55:58.293 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:55:58Z" } 2022-07-18T03:55:58.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:55:58.295 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:55:58.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:55:58.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:03.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:03.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:03.298 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:03.347 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:03.348 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:03.434 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:03.434 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:03.435 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:03.435 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:03.435 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:03.436 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:03.436 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:03.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:03.436 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:03.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:03.437 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:03.437 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:03.437 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:03.438 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:03.438 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:03.438 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:03.439 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:03.439 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:03.439 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:03.439 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:03.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:03.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:03.440 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:03.440 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:03.441 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:03.441 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:03.441 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:03.442 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:03.442 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:03.442 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:03.442 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:03.443 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:03.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:03.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:03.443 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:03.444 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:03.444 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:03.444 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:03.444 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:03.445 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:03.445 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:03.445 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:03.445 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:03.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:03.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:03.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:03.447 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:03.447 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:03.447 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:03.448 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:03.448 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:03.448 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:03.449 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:03.449 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:03.449 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:03.450 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:03.450 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:03.450 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:03.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:03.451 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:03.451 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:03.451 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:03.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:03.452 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:03Z" 2022-07-18T03:56:03.452 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:03.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:03Z" } 2022-07-18T03:56:03.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:03.455 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:03.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:03.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:08.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:08.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:08.460 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:08.514 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:08.515 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:08.588 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:08.589 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:08.589 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:08.589 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:08.589 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:08.590 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:08.590 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:08.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:08.591 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:08.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:08.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:08.591 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:08.592 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:08.592 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:08.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:08.592 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:08.593 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:08.593 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:08.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:08.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:08.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:08.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:08.594 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:08.594 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:08.595 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:08.595 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:08.595 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:08.595 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:08.596 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:08.596 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:08.596 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:08.597 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:08.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:08.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:08.597 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:08.598 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:08.598 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:08.598 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:08.598 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:08.599 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:08.599 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:08.599 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:08.599 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:08.600 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:08.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:08.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:08.600 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:08.601 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:08.601 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:08.601 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:08.601 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:08.602 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:08.602 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:08.602 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:08.602 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:08.603 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:08.603 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:08.603 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:08.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:08.604 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:08.604 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:08.604 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:08.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:08.605 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:08Z" 2022-07-18T03:56:08.605 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:08.606 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:08Z" } 2022-07-18T03:56:08.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:08.608 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:08.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:08.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:13.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:13.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:13.611 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:13.667 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:13.667 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:13.736 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:13.736 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:13.736 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:13.737 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:13.737 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:13.737 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:13.738 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:13.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:13.738 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:13.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:13.739 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:13.739 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:13.739 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:13.739 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:13.740 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:13.740 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:13.740 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:13.740 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:13.741 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:13.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:13.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:13.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:13.742 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:13.742 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:13.742 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:13.743 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:13.743 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:13.743 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:13.744 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:13.744 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:13.744 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:13.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:13.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:13.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:13.745 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:13.745 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:13.746 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:13.746 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:13.746 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:13.747 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:13.747 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:13.747 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:13.747 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:13.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:13.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:13.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:13.748 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:13.749 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:13.749 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:13.749 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:13.749 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:13.750 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:13.750 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:13.750 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:13.750 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:13.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:13.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:13.751 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:13.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:13.752 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:13.752 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:13.752 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:13.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:13.753 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:13Z" 2022-07-18T03:56:13.753 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:13.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:13Z" } 2022-07-18T03:56:13.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:13.755 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:13.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:13.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:18.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:18.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:18.759 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:18.810 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:18.811 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:18.869 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:18.870 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:18.870 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:18.870 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:18.871 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:18.871 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:18.871 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:18.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:18.872 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:18.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:18.872 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:18.873 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:18.873 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:18.873 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:18.874 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:18.874 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:18.874 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:18.874 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:18.875 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:18.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:18.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:18.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:18.876 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:18.877 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:18.877 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:18.878 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:18.879 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:18.879 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:18.880 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:18.880 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:18.880 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:18.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:18.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:18.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:18.881 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:18.882 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:18.882 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:18.882 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:18.883 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:18.883 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:18.883 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:18.883 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:18.884 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:18.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:18.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:18.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:18.887 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:18.887 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:18.887 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:18.887 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:18.888 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:18.888 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:18.888 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:18.888 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:18.889 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:18.889 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:18.889 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:18.889 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:18.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:18.890 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:18.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:18.890 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:18.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:18.891 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:18Z" 2022-07-18T03:56:18.891 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:18.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:18Z" } 2022-07-18T03:56:18.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:18.894 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:18.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:18.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:23.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:23.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:23.897 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:23.945 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:23.946 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:24.031 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:24.032 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:24.032 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:24.032 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:24.033 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:24.033 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:24.033 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:24.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:24.034 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:24.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:24.034 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:24.035 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:24.035 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:24.035 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:24.035 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:24.036 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:24.036 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:24.036 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:24.036 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:24.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:24.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:24.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:24.038 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:24.038 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:24.039 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:24.039 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:24.039 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:24.039 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:24.040 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:24.040 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:24.041 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:24.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:24.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:24.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:24.042 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:24.042 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:24.042 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:24.043 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:24.043 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:24.043 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:24.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:24.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:24.044 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:24.044 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:24.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:24.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:24.045 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:24.046 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:24.046 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:24.046 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:24.046 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:24.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:24.047 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:24.048 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:24.048 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:24.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:24.049 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:24.049 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:24.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:24.049 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:24.050 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:24.050 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:24.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:24.051 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:24Z" 2022-07-18T03:56:24.052 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:24.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:24Z" } 2022-07-18T03:56:24.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:24.054 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:24.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:24.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:29.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:29.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:29.058 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:29.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:29.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:29.188 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:29.188 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:29.188 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:29.189 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:29.189 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:29.189 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:29.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:29.191 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:29.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:29.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:29.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:29.192 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:29.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:29.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:29.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:29.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:29.194 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:29.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:29.194 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:29.195 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:29.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:29.195 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:29.195 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:29.196 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:29.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:29.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:29.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:29.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:29.198 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:29.198 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:29.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:29.199 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:29.199 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:29.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:29.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:29.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:29.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:29.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:29.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:29.201 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:29.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:29.201 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:29.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:29.202 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:29.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:29.202 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:29.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:29.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:29.203 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:29.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:29.204 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:29.204 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:29.204 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:29.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:29.205 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:29Z" 2022-07-18T03:56:29.205 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:29.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:29Z" } 2022-07-18T03:56:29.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:29.207 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:29.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:29.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:34.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:34.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:34.212 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:34.267 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:34.268 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:34.330 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:34.331 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:34.331 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:34.331 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:34.332 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:34.332 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:34.332 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:34.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:34.333 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:34.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:34.334 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:34.334 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:34.334 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:34.334 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:34.335 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:34.335 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:34.335 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:34.336 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:34.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:34.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:34.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:34.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:34.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:34.338 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:34.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:34.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:34.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:34.339 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:34.339 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:34.339 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:34.340 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:34.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:34.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:34.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:34.341 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:34.341 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:34.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:34.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:34.342 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:34.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:34.342 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:34.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:34.343 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:34.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:34.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:34.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:34.344 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:34.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:34.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:34.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:34.345 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:34.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:34.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:34.345 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:34.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:34.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:34.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:34.346 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:34.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:34.347 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:34.347 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:34.347 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:34.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:34.348 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:34Z" 2022-07-18T03:56:34.348 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:34.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:34Z" } 2022-07-18T03:56:34.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:34.351 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:34.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:34.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:39.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:39.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:39.354 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:39.410 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:39.410 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:39.486 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:39.487 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:39.487 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:39.487 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:39.487 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:39.488 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:39.488 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:39.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:39.489 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:39.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:39.489 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:39.490 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:39.490 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:39.490 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:39.491 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:39.491 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:39.491 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:39.491 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:39.492 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:39.492 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:39.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:39.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:39.493 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:39.493 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:39.494 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:39.494 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:39.494 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:39.495 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:39.495 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:39.495 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:39.496 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:39.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:39.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:39.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:39.497 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:39.497 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:39.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:39.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:39.498 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:39.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:39.499 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:39.499 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:39.499 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:39.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:39.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:39.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:39.501 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:39.501 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:39.501 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:39.502 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:39.502 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:39.502 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:39.502 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:39.503 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:39.503 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:39.503 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:39.504 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:39.504 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:39.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:39.505 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:39.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:39.506 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:39.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:39.506 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:39Z" 2022-07-18T03:56:39.507 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:39.509 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:39Z" } 2022-07-18T03:56:39.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:39.511 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:39.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:39.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:44.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:44.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:44.516 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:44.569 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:44.569 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:44.636 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:44.636 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:44.636 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:44.637 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:44.637 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:44.638 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:44.638 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:44.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:44.639 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:44.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:44.639 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:44.640 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:44.640 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:44.641 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:44.641 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:44.641 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:44.642 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:44.642 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:44.642 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:44.642 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:44.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:44.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:44.643 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:44.644 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:44.644 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:44.645 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:44.645 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:44.646 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:44.646 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:44.647 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:44.647 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:44.648 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:44.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:44.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:44.649 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:44.649 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:44.650 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:44.650 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:44.651 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:44.651 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:44.651 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:44.652 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:44.652 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:44.652 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:44.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:44.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:44.653 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:44.653 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:44.653 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:44.654 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:44.654 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:44.654 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:44.654 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:44.655 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:44.655 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:44.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:44.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:44.656 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:44.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:44.656 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:44.656 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:44.657 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:44.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:44.657 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:44Z" 2022-07-18T03:56:44.657 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:44.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:44Z" } 2022-07-18T03:56:44.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:44.660 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:44.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:44.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:49.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:49.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:49.664 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:49.719 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:49.720 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:49.793 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:49.793 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:49.794 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:49.794 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:49.794 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:49.794 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:49.795 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:49.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:49.795 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:49.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:49.796 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:49.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:49.797 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:49.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:49.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:49.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:49.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:49.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:49.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:49.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:49.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:49.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:49.799 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:49.799 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:49.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:49.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:49.800 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:49.801 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:49.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:49.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:49.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:49.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:49.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:49.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:49.803 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:49.803 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:49.803 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:49.803 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:49.804 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:49.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:49.804 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:49.804 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:49.805 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:49.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:49.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:49.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:49.806 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:49.806 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:49.807 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:49.807 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:49.807 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:49.807 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:49.808 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:49.808 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:49.808 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:49.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:49.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:49.810 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:49.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:49.811 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:49.811 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:49.811 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:49.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:49.812 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:49Z" 2022-07-18T03:56:49.812 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:49.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:49Z" } 2022-07-18T03:56:49.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:49.815 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:49.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:49.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:54.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:54.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:54.819 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:56:54.874 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:56:54.875 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:56:54.965 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:56:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:56:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:56:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:56:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:56:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:56:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:54.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:56:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:56:54.968 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:56:54.969 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:56:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:56:54.970 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:56:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:56:54.972 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:56:54.973 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:54.974 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:56:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:56:54.975 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:56:54.976 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:54.977 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:54.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:54.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:56:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:56:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:56:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:56:54.978 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:56:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:56:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:56:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:56:54.979 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:56:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:56:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:56:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:56:54.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:54.981 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:56:54.981 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:56:54.981 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:56:54.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:56:54.982 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:56:54Z" 2022-07-18T03:56:54.982 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:56:54.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:56:54Z" } 2022-07-18T03:56:54.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:56:54.985 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:56:54.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:56:54.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:56:59.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:59.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:56:59.988 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:00.046 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:00.046 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:00.127 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:00.128 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:00.128 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:00.128 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:00.128 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:00.129 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:00.129 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:00.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:00.129 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:00.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:00.130 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:00.130 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:00.131 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:00.131 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:00.131 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:00.131 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:00.132 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:00.132 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:00.132 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:00.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:00.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:00.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:00.133 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:00.134 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:00.134 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:00.134 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:00.135 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:00.135 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:00.135 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:00.135 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:00.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:00.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:00.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:00.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:00.137 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:00.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:00.137 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:00.137 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:00.138 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:00.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:00.138 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:00.138 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:00.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:00.140 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:00.140 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:00.140 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:00.140 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:00.141 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:00.141 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:00.141 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:00.141 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:00.142 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:00.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:00.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:00.142 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:00.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:00.143 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:00.143 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:00.143 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:00.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:00.144 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:00Z" 2022-07-18T03:57:00.145 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:00.146 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:00Z" } 2022-07-18T03:57:00.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:00.147 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:00.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:00.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:05.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:05.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:05.151 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:05.206 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:05.206 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:05.276 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:05.276 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:05.276 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:05.277 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:05.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:05.279 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:05.280 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:05.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:05.282 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:05.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:05.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:05.285 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:05.286 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:05.287 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:05.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:05.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:05.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:05.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:05.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:05.290 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:05.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:05.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:05.291 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:05.292 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:05.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:05.292 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:05Z" 2022-07-18T03:57:05.292 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:05.294 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:05Z" } 2022-07-18T03:57:05.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:05.295 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:05.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:05.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:10.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:10.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:10.298 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:10.354 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:10.355 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:10.427 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:10.427 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:10.427 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:10.428 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:10.428 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:10.428 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:10.429 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:10.430 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:10.430 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:10.430 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:10.431 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:10.431 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:10.431 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:10.431 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:10.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:10.433 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:10.433 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:10.433 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:10.433 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:10.434 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:10.435 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:10.436 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:10.436 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:10.436 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:10.436 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:10.437 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:10.437 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:10.437 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:10.437 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:10.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:10.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:10.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:10.438 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:10.439 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:10.439 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:10.439 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:10.439 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:10.440 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:10.440 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:10.440 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:10.440 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:10.441 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:10.441 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:10.441 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:10.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:10.442 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:10.442 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:10.442 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:10.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:10.443 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:10Z" 2022-07-18T03:57:10.443 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:10.444 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:10Z" } 2022-07-18T03:57:10.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:10.446 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:10.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:10.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:15.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:15.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:15.449 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:15.505 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:15.506 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:15.568 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:15.568 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:15.569 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:15.569 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:15.570 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:15.570 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:15.570 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:15.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:15.571 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:15.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:15.571 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:15.571 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:15.572 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:15.572 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:15.572 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:15.572 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:15.573 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:15.573 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:15.573 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:15.573 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:15.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:15.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:15.574 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:15.574 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:15.574 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:15.575 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:15.575 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:15.575 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:15.575 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:15.576 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:15.576 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:15.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:15.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:15.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:15.577 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:15.577 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:15.577 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:15.578 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:15.578 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:15.578 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:15.579 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:15.579 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:15.579 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:15.579 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:15.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:15.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:15.580 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:15.580 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:15.581 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:15.581 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:15.581 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:15.581 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:15.582 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:15.582 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:15.582 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:15.582 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:15.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:15.583 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:15.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:15.583 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:15.584 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:15.584 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:15.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:15.584 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:15Z" 2022-07-18T03:57:15.585 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:15.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:15Z" } 2022-07-18T03:57:15.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:15.587 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:15.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:15.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:20.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:20.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:20.593 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:20.650 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:20.650 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:20.729 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:20.730 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:20.730 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:20.730 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:20.731 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:20.731 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:20.731 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:20.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:20.732 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:20.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:20.732 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:20.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:20.733 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:20.733 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:20.733 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:20.734 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:20.734 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:20.734 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:20.734 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:20.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:20.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:20.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:20.735 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:20.736 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:20.736 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:20.736 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:20.736 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:20.737 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:20.737 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:20.737 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:20.737 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:20.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:20.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:20.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:20.739 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:20.739 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:20.739 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:20.740 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:20.740 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:20.740 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:20.740 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:20.741 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:20.741 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:20.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:20.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:20.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:20.742 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:20.742 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:20.742 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:20.743 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:20.743 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:20.743 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:20.743 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:20.744 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:20.744 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:20.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:20.745 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:20.745 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:20.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:20.745 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:20.746 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:20.746 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:20.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:20.746 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:20Z" 2022-07-18T03:57:20.746 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:20.748 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:20Z" } 2022-07-18T03:57:20.749 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:20.749 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:20.750 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:20.750 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:25.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:25.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:25.753 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:25.810 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:25.811 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:25.873 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:25.873 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:25.874 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:25.874 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:25.875 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:25.875 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:25.875 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:25.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:25.876 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:25.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:25.876 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:25.876 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:25.877 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:25.877 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:25.877 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:25.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:25.878 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:25.878 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:25.878 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:25.879 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:25.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:25.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:25.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:25.880 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:25.880 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:25.880 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:25.880 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:25.881 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:25.881 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:25.881 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:25.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:25.882 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:25.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:25.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:25.882 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:25.883 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:25.883 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:25.883 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:25.883 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:25.884 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:25.884 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:25.884 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:25.885 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:25.885 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:25.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:25.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:25.886 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:25.886 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:25.886 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:25.886 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:25.887 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:25.887 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:25.887 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:25.888 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:25.888 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:25.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:25.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:25.889 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:25.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:25.889 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:25.889 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:25.890 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:25.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:25.890 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:25Z" 2022-07-18T03:57:25.890 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:25.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:25Z" } 2022-07-18T03:57:25.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:25.893 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:25.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:25.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:30.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:30.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:30.898 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:30.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:30.947 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:31.023 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:31.023 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:31.024 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:31.024 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:31.024 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:31.025 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:31.025 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:31.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:31.025 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:31.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:31.026 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:31.026 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:31.027 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:31.027 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:31.027 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:31.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:31.028 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:31.028 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:31.028 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:31.028 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:31.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:31.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:31.029 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:31.029 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:31.030 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:31.030 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:31.030 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:31.030 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:31.031 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:31.031 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:31.031 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:31.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:31.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:31.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:31.032 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:31.032 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:31.033 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:31.033 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:31.033 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:31.033 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:31.034 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:31.034 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:31.034 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:31.034 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:31.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:31.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:31.035 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:31.035 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:31.036 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:31.036 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:31.036 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:31.036 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:31.037 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:31.037 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:31.038 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:31.038 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:31.038 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:31.038 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:31.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:31.039 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:31.039 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:31.039 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:31.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:31.040 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:31Z" 2022-07-18T03:57:31.040 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:31.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:31Z" } 2022-07-18T03:57:31.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:31.043 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:31.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:31.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:36.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:36.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:36.046 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:36.097 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:36.097 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:36.187 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:36.188 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:36.188 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:36.188 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:36.189 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:36.189 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:36.189 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:36.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:36.190 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:36.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:36.191 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:36.191 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:36.191 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:36.191 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:36.192 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:36.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:36.192 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:36.193 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:36.193 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:36.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:36.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:36.195 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:36.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:36.195 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:36.195 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:36.196 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:36.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:36.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:36.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:36.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:36.197 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:36.197 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:36.198 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:36.198 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:36.198 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:36.199 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:36.199 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:36.199 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:36.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:36.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:36.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:36.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:36.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:36.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:36.202 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:36.202 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:36.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:36.203 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:36.203 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:36.203 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:36.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:36.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:36.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:36.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:36.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:36.205 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:36.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:36.205 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:36.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:36.206 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:36Z" 2022-07-18T03:57:36.206 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:36.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:36Z" } 2022-07-18T03:57:36.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:36.209 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:36.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:36.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:41.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:41.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:41.215 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:41.270 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:41.271 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:41.334 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:41.335 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:41.335 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:41.335 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:41.336 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:41.336 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:41.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:41.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:41.337 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:41.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:41.337 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:41.337 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:41.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:41.338 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:41.338 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:41.338 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:41.339 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:41.339 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:41.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:41.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:41.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:41.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:41.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:41.341 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:41.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:41.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:41.341 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:41.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:41.342 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:41.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:41.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:41.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:41.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:41.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:41.343 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:41.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:41.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:41.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:41.344 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:41.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:41.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:41.346 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:41.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:41.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:41.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:41.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:41.347 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:41.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:41.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:41.348 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:41.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:41.348 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:41.348 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:41.349 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:41.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:41.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:41.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:41.350 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:41.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:41.350 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:41.350 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:41.351 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:41.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:41.351 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:41Z" 2022-07-18T03:57:41.351 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:41.353 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:41Z" } 2022-07-18T03:57:41.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:41.354 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:41.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:41.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:46.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:46.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:46.358 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:46.413 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:46.414 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:46.492 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:46.492 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:46.493 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:46.493 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:46.493 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:46.493 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:46.494 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:46.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:46.494 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:46.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:46.495 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:46.495 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:46.495 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:46.495 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:46.496 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:46.496 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:46.496 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:46.497 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:46.497 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:46.497 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:46.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:46.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:46.498 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:46.498 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:46.498 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:46.499 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:46.499 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:46.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:46.499 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:46.500 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:46.500 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:46.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:46.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:46.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:46.501 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:46.501 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:46.501 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:46.502 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:46.502 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:46.502 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:46.503 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:46.503 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:46.503 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:46.503 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:46.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:46.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:46.504 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:46.504 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:46.504 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:46.505 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:46.505 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:46.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:46.505 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:46.506 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:46.506 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:46.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:46.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:46.507 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:46.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:46.507 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:46.507 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:46.508 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:46.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:46.508 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:46Z" 2022-07-18T03:57:46.508 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:46.510 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:46Z" } 2022-07-18T03:57:46.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:46.511 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:46.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:46.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:51.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:51.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:51.515 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:51.570 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:51.570 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:51.633 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:51.633 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:51.634 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:51.634 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:51.634 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:51.635 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:51.635 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:51.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:51.635 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:51.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:51.636 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:51.636 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:51.636 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:51.637 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:51.637 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:51.637 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:51.637 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:51.638 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:51.638 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:51.638 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:51.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:51.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:51.639 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:51.639 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:51.639 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:51.640 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:51.640 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:51.640 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:51.640 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:51.641 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:51.641 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:51.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:51.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:51.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:51.642 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:51.642 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:51.642 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:51.642 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:51.643 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:51.643 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:51.643 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:51.643 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:51.644 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:51.644 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:51.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:51.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:51.645 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:51.645 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:51.645 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:51.646 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:51.646 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:51.646 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:51.646 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:51.647 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:51.647 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:51.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:51.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:51.648 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:51.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:51.648 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:51.649 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:51.649 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:51.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:51.649 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:51Z" 2022-07-18T03:57:51.650 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:51.651 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:51Z" } 2022-07-18T03:57:51.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:51.652 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:51.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:51.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:57:56.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:56.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:57:56.656 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:57:56.708 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:57:56.708 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:57:56.784 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:57:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:57:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:57:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:57:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:57:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:57:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:57:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:57:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:57:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:57:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:57:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:57:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:57:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:57:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:57:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:57:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:57:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:57:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:57:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:57:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:57:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:57:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:57:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:57:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:57:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:57:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:57:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:57:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:57:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:57:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:57:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:57:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:57:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:57:56Z" 2022-07-18T03:57:56.801 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:57:56.802 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:57:56Z" } 2022-07-18T03:57:56.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:57:56.803 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:57:56.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:57:56.804 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:01.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:01.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:01.806 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:01.856 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:01.857 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:01.924 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:01.924 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:01.925 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:01.925 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:01.926 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:01.926 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:01.926 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:01.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:01.927 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:01.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:01.927 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:01.927 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:01.928 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:01.928 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:01.928 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:01.929 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:01.929 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:01.929 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:01.929 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:01.930 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:01.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:01.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:01.930 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:01.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:01.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:01.931 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:01.931 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:01.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:01.932 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:01.932 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:01.933 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:01.933 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:01.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:01.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:01.933 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:01.934 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:01.934 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:01.934 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:58:01.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:01.935 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:01.935 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:01.935 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:58:01.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:01.936 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:01.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:01.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:01.937 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:01.937 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:01.937 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:01.937 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:01.938 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:01.938 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:01.938 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:01.939 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:01.939 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:01.939 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:01.939 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:01.940 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:01.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:01.940 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:01.940 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:01.941 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:01.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:01.941 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:01Z" 2022-07-18T03:58:01.941 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:01.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:01Z" } 2022-07-18T03:58:01.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:01.945 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:01.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:01.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:06.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:06.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:06.949 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:06.997 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:06.998 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:07.072 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:07.073 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:07.073 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:07.074 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:07.074 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:07.074 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:07.074 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:07.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:07.075 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:07.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:07.075 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:07.076 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:07.076 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:07.076 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:07.076 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:07.077 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:07.077 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:07.077 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:07.077 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:07.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:07.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:07.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:07.078 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:07.079 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:07.079 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:07.079 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:07.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:07.080 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:07.080 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:07.080 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:07.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:07.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:07.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:07.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:07.081 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:07.082 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:07.082 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:07.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:58:07.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:07.083 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:07.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:07.083 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:58:07.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:07.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:07.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:07.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:07.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:07.085 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:07.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:07.085 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:07.085 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:07.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:07.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:07.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:07.086 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:07.086 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:07.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:07.087 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:07.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:07.088 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:07.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:07.088 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:07.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:07.088 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:07Z" 2022-07-18T03:58:07.089 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:07.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:07Z" } 2022-07-18T03:58:07.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:07.091 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:07.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:07.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:12.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:12.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:12.094 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:12.150 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:12.150 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:12.222 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:12.223 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:12.223 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:12.223 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:12.224 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:12.224 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:12.224 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:12.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:12.225 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:12.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:12.225 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:12.225 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:12.226 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:12.226 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:12.226 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:12.227 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:12.227 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:12.227 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:12.227 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:12.228 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:12.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:12.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:12.228 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:12.229 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:12.229 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:12.229 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:12.229 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:12.230 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:12.230 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:12.230 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:12.230 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:12.231 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:12.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:12.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:12.231 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:12.232 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:12.232 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:12.232 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:58:12.232 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:12.233 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:12.233 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:12.233 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:58:12.233 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:12.234 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:12.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:12.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:12.234 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:12.235 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:12.235 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:12.235 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:12.235 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:12.236 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:12.236 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:12.236 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:12.236 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:12.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:12.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:12.237 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:12.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:12.238 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:12.238 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:12.238 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:12.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:12.239 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:12Z" 2022-07-18T03:58:12.239 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:12.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:12Z" } 2022-07-18T03:58:12.241 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:12.241 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:12.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:12.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:17.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:17.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:17.245 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:17.301 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:17.301 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:17.372 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:17.372 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:17.373 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:17.373 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:17.373 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:17.373 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:17.374 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:17.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:17.374 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:17.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:17.375 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:17.375 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:17.375 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:17.376 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:17.376 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:17.376 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:17.376 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:17.377 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:17.377 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:17.377 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:17.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:17.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:17.378 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:17.378 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:17.379 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:17.379 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:17.379 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:17.379 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:17.380 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:17.380 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:17.380 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:17.380 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:17.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:17.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:17.381 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:17.381 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:17.382 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:17.382 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:58:17.382 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:17.382 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:17.383 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:17.383 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:58:17.383 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:17.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:17.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:17.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:17.384 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:17.384 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:17.385 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:17.385 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:17.385 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:17.385 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:17.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:17.386 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:17.386 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:17.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:17.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:17.387 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:17.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:17.387 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:17.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:17.388 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:17.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:17.388 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:17Z" 2022-07-18T03:58:17.388 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:17.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:17Z" } 2022-07-18T03:58:17.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:17.391 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:17.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:17.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:22.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:22.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:22.395 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:22.450 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:22.451 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:22.532 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:58:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:58:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:22Z" 2022-07-18T03:58:22.548 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:22.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:22Z" } 2022-07-18T03:58:22.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:22.551 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:22.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:22.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:27.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:27.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:27.554 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:27.612 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:27.612 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:27.693 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:27.693 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:27.694 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:27.695 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:27.697 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:27.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:27.699 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:27.700 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:27.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:27.702 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116402.849523_272.1", 2022-07-18T03:58:27.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:27.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:27.703 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:27.703 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:53:22.849523Z", 2022-07-18T03:58:27.704 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:27.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:27.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:27.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:27.705 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:27.705 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:27.705 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:27.705 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:27.706 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:27.706 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:27.706 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:27.706 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:27.707 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:27.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:27.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:27.707 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:27.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:27.708 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:27.708 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:27.708 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:27.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:27.709 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:27Z" 2022-07-18T03:58:27.709 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:27.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116402.849523_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:53:22.849523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:27Z" } 2022-07-18T03:58:27.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116402.849523_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:53:22.849523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:27.712 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:27.712 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116402.849523_272.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:27.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:32.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2022-07-18T03:58:32.721 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-42 2022-07-18T03:58:32.722 DEBUG:boto:path=/mlvank-42/ 2022-07-18T03:58:32.723 DEBUG:boto:auth_path=/mlvank-42/ 2022-07-18T03:58:32.724 DEBUG:boto:Method: PUT 2022-07-18T03:58:32.724 DEBUG:boto:Path: /mlvank-42/ 2022-07-18T03:58:32.725 DEBUG:boto:Data: 2022-07-18T03:58:32.725 DEBUG:boto:Headers: {} 2022-07-18T03:58:32.726 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:58:32.726 DEBUG:boto:Port: 8000 2022-07-18T03:58:32.727 DEBUG:boto:Params: {} 2022-07-18T03:58:32.728 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:58:32.729 DEBUG:boto:Token: None 2022-07-18T03:58:32.730 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:58:32 GMT /mlvank-42/ 2022-07-18T03:58:32.730 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:Ras6bTEj+U0i92M2JYLY9IN5dUo= 2022-07-18T03:58:32.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:Ras6bTEj+U0i92M2JYLY9IN5dUo=', 'Content-Length': '0'} 2022-07-18T03:58:32.760 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a059dc2a55a5d3ce-0062d4da68-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.761 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=mlvank-43 2022-07-18T03:58:32.761 DEBUG:boto:path=/mlvank-43/ 2022-07-18T03:58:32.762 DEBUG:boto:auth_path=/mlvank-43/ 2022-07-18T03:58:32.762 DEBUG:boto:Method: PUT 2022-07-18T03:58:32.763 DEBUG:boto:Path: /mlvank-43/ 2022-07-18T03:58:32.763 DEBUG:boto:Data: 2022-07-18T03:58:32.764 DEBUG:boto:Headers: {} 2022-07-18T03:58:32.764 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:58:32.765 DEBUG:boto:Port: 8000 2022-07-18T03:58:32.766 DEBUG:boto:Params: {} 2022-07-18T03:58:32.766 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T03:58:32.767 DEBUG:boto:Token: None 2022-07-18T03:58:32.767 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:58:32 GMT /mlvank-43/ 2022-07-18T03:58:32.768 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:J5c590fzUw4ro78Lv/2vzv9czTA= 2022-07-18T03:58:32.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:J5c590fzUw4ro78Lv/2vzv9czTA=', 'Content-Length': '0'} 2022-07-18T03:58:32.818 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009fb109c9c131a1bc-0062d4da68-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.819 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=mlvank-44 2022-07-18T03:58:32.819 DEBUG:boto:path=/mlvank-44/ 2022-07-18T03:58:32.820 DEBUG:boto:auth_path=/mlvank-44/ 2022-07-18T03:58:32.820 DEBUG:boto:Method: PUT 2022-07-18T03:58:32.821 DEBUG:boto:Path: /mlvank-44/ 2022-07-18T03:58:32.821 DEBUG:boto:Data: 2022-07-18T03:58:32.822 DEBUG:boto:Headers: {} 2022-07-18T03:58:32.823 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:58:32.823 DEBUG:boto:Port: 8001 2022-07-18T03:58:32.823 DEBUG:boto:Params: {} 2022-07-18T03:58:32.824 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-07-18T03:58:32.825 DEBUG:boto:Token: None 2022-07-18T03:58:32.825 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 03:58:32 GMT /mlvank-44/ 2022-07-18T03:58:32.826 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:F/fAqAQm/qtrPj1C1yAReymo+nE= 2022-07-18T03:58:32.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:F/fAqAQm/qtrPj1C1yAReymo+nE=', 'Content-Length': '0'} 2022-07-18T03:58:32.872 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086fcac465133f43c-0062d4da68-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.874 DEBUG:boto:path=/mlvank-42/ 2022-07-18T03:58:32.874 DEBUG:boto:auth_path=/mlvank-42/ 2022-07-18T03:58:32.875 DEBUG:boto:Method: HEAD 2022-07-18T03:58:32.876 DEBUG:boto:Path: /mlvank-42/ 2022-07-18T03:58:32.876 DEBUG:boto:Data: 2022-07-18T03:58:32.877 DEBUG:boto:Headers: {} 2022-07-18T03:58:32.877 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:58:32.878 DEBUG:boto:Port: 8000 2022-07-18T03:58:32.879 DEBUG:boto:Params: {} 2022-07-18T03:58:32.880 DEBUG:boto:Token: None 2022-07-18T03:58:32.881 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:58:32 GMT /mlvank-42/ 2022-07-18T03:58:32.881 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:yMHzYZ8pbtXs+ktCC6fDPGszqxo= 2022-07-18T03:58:32.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:yMHzYZ8pbtXs+ktCC6fDPGszqxo=', 'Content-Length': '0'} 2022-07-18T03:58:32.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cd146694b887155-0062d4da68-10f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.887 DEBUG:boto:path=/mlvank-42/key 2022-07-18T03:58:32.888 DEBUG:boto:auth_path=/mlvank-42/key 2022-07-18T03:58:32.888 DEBUG:boto:Method: PUT 2022-07-18T03:58:32.889 DEBUG:boto:Path: /mlvank-42/key 2022-07-18T03:58:32.890 DEBUG:boto:Data: 2022-07-18T03:58:32.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-07-18T03:58:32.891 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T03:58:32.891 DEBUG:boto:Port: 8000 2022-07-18T03:58:32.892 DEBUG:boto:Params: {} 2022-07-18T03:58:32.892 DEBUG:boto:Token: None 2022-07-18T03:58:32.893 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 18 Jul 2022 03:58:32 GMT /mlvank-42/key 2022-07-18T03:58:32.894 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:chlMYZoj+xwBspjdohWye7ITybM= 2022-07-18T03:58:32.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:chlMYZoj+xwBspjdohWye7ITybM='} 2022-07-18T03:58:32.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cafe2f96bfd725b5-0062d4da68-10f9-test-zone1'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.903 DEBUG:boto:path=/mlvank-43/ 2022-07-18T03:58:32.904 DEBUG:boto:auth_path=/mlvank-43/ 2022-07-18T03:58:32.905 DEBUG:boto:Method: HEAD 2022-07-18T03:58:32.906 DEBUG:boto:Path: /mlvank-43/ 2022-07-18T03:58:32.906 DEBUG:boto:Data: 2022-07-18T03:58:32.907 DEBUG:boto:Headers: {} 2022-07-18T03:58:32.908 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:58:32.909 DEBUG:boto:Port: 8000 2022-07-18T03:58:32.910 DEBUG:boto:Params: {} 2022-07-18T03:58:32.911 DEBUG:boto:Token: None 2022-07-18T03:58:32.912 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:58:32 GMT /mlvank-43/ 2022-07-18T03:58:32.913 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:JnvON2kumjPTNwkkrz9/KwOkz/4= 2022-07-18T03:58:32.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:JnvON2kumjPTNwkkrz9/KwOkz/4=', 'Content-Length': '0'} 2022-07-18T03:58:32.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb61e64ec32ad3d3-0062d4da68-10e2-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.926 DEBUG:boto:path=/mlvank-43/key 2022-07-18T03:58:32.927 DEBUG:boto:auth_path=/mlvank-43/key 2022-07-18T03:58:32.928 DEBUG:boto:Method: PUT 2022-07-18T03:58:32.929 DEBUG:boto:Path: /mlvank-43/key 2022-07-18T03:58:32.930 DEBUG:boto:Data: 2022-07-18T03:58:32.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-07-18T03:58:32.931 DEBUG:boto:Host: smithi159.front.sepia.ceph.com:8000 2022-07-18T03:58:32.933 DEBUG:boto:Port: 8000 2022-07-18T03:58:32.934 DEBUG:boto:Params: {} 2022-07-18T03:58:32.935 DEBUG:boto:Token: None 2022-07-18T03:58:32.936 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 18 Jul 2022 03:58:32 GMT /mlvank-43/key 2022-07-18T03:58:32.937 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:/mppCO3AKIwMZ3Pa1QOpG4WE9jw= 2022-07-18T03:58:32.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:/mppCO3AKIwMZ3Pa1QOpG4WE9jw='} 2022-07-18T03:58:32.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f8bf43dbc71bc86-0062d4da68-10e2-test-zone2'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.963 DEBUG:boto:path=/mlvank-44/ 2022-07-18T03:58:32.964 DEBUG:boto:auth_path=/mlvank-44/ 2022-07-18T03:58:32.965 DEBUG:boto:Method: HEAD 2022-07-18T03:58:32.966 DEBUG:boto:Path: /mlvank-44/ 2022-07-18T03:58:32.967 DEBUG:boto:Data: 2022-07-18T03:58:32.968 DEBUG:boto:Headers: {} 2022-07-18T03:58:32.969 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:58:32.970 DEBUG:boto:Port: 8001 2022-07-18T03:58:32.971 DEBUG:boto:Params: {} 2022-07-18T03:58:32.972 DEBUG:boto:Token: None 2022-07-18T03:58:32.973 DEBUG:boto:StringToSign: HEAD Mon, 18 Jul 2022 03:58:32 GMT /mlvank-44/ 2022-07-18T03:58:32.973 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:yuqI+AJHWlBu2c6a/Hp2vT1dQPU= 2022-07-18T03:58:32.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:yuqI+AJHWlBu2c6a/Hp2vT1dQPU=', 'Content-Length': '0'} 2022-07-18T03:58:32.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c70b63625d3508c-0062d4da68-10f7-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.977 DEBUG:boto:path=/mlvank-44/key 2022-07-18T03:58:32.978 DEBUG:boto:auth_path=/mlvank-44/key 2022-07-18T03:58:32.978 DEBUG:boto:Method: PUT 2022-07-18T03:58:32.979 DEBUG:boto:Path: /mlvank-44/key 2022-07-18T03:58:32.979 DEBUG:boto:Data: 2022-07-18T03:58:32.980 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-07-18T03:58:32.981 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8001 2022-07-18T03:58:32.981 DEBUG:boto:Port: 8001 2022-07-18T03:58:32.982 DEBUG:boto:Params: {} 2022-07-18T03:58:32.982 DEBUG:boto:Token: None 2022-07-18T03:58:32.983 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 18 Jul 2022 03:58:32 GMT /mlvank-44/key 2022-07-18T03:58:32.983 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:udlVUO0TSp+jOLeBTevkZhLes94= 2022-07-18T03:58:32.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 03:58:32 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:udlVUO0TSp+jOLeBTevkZhLes94='} 2022-07-18T03:58:32.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3e1d7b050ab1458-0062d4da68-10f7-test-zone3'), ('Date', 'Mon, 18 Jul 2022 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T03:58:32.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:32.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:32.994 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:33.043 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:33.043 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:33.090 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T03:58:33.102 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T03:58:33.102 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:58:33.113 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:33.114 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T03:58:33.114 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:58:33.114 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:58:33.114 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T03:58:33.115 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T03:58:33.115 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:58:33.115 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:58:33.116 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:58:33.116 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:58:32.851438Z" 2022-07-18T03:58:33.116 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T03:58:33.116 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T03:58:33.117 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T03:58:33.117 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T03:58:33.117 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T03:58:33.117 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T03:58:33.119 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:33.119 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T03:58:33.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:33.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:33.121 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:33.169 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:33.169 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:33.243 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:33.243 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:33.243 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:33.244 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:33.244 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:33.244 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:33.244 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:33.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:33.245 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:33.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:33.246 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:33.246 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:33.246 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:33.246 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:33.247 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:33.247 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:33.247 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:33.247 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:33.248 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:33.248 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:33.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:33.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:33.249 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:33.249 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:33.249 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:33.249 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:33.250 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:33.250 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:33.250 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:33.251 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:33.251 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:33.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:33.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:33.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:33.252 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:33.252 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:33.252 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:33.253 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.746179_277.1", 2022-07-18T03:58:33.253 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:33.253 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:33.253 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:33.254 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.746179Z", 2022-07-18T03:58:33.254 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:33.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:33.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:33.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:33.255 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:33.255 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:33.256 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:33.256 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:33.256 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:33.256 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:33.257 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:33.257 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:33.257 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:33.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:33.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:33.258 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:33.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:33.258 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:33.259 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:33.259 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:33.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:33.259 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:33Z" 2022-07-18T03:58:33.260 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:33.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.746179_277.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.746179Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:33Z" } 2022-07-18T03:58:33.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.746179_277.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.746179Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:33.262 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:33.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.746179_277.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:33.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658116712.851438_288.1 target=1_1658116712.746179_277.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:38.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:38.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:38.266 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:38.322 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:38.323 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:38.396 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:38.396 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:38.397 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:38.397 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:38.397 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:38.398 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:38.398 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:38.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:38.398 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:38.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:38.399 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:38.399 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:38.400 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:38.400 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:38.400 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:38.400 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:38.401 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:38.401 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:38.401 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:38.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:38.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:38.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:38.402 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:38.403 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:38.403 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:38.403 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:38.403 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:38.404 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:38.404 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:38.404 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:38.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:38.405 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:38.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:38.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:38.405 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:38.406 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:38.406 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:38.406 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:58:38.407 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:38.407 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:38.407 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:38.407 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:58:38.408 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:38.408 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:38.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:38.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:38.409 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:38.409 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:38.409 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:38.410 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:38.410 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:38.410 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:38.410 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:38.411 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:38.411 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:38.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:38.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:38.412 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:38.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:38.412 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:38.413 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:38.413 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:38.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:38.413 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:38Z" 2022-07-18T03:58:38.414 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:38.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:38Z" } 2022-07-18T03:58:38.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:38.416 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:38.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:38.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:43.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:43.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:43.421 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:43.470 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:43.471 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:43.542 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:43.543 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:43.543 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:43.543 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:43.543 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:43.544 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:43.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:43.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:43.545 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:43.546 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:43.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:43.548 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:43.549 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:43.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:43.551 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:58:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:43.552 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:58:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:43.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:43.554 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:43.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:43.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:43.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:43.558 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:43Z" 2022-07-18T03:58:43.558 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:43.560 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:43Z" } 2022-07-18T03:58:43.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:43.561 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:43.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:43.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:48.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:48.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:48.565 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:48.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:48.621 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:48.689 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:58:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:58:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:48Z" 2022-07-18T03:58:48.709 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:48.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:48Z" } 2022-07-18T03:58:48.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:48.712 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:48.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:48.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:53.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:53.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:53.717 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:53.765 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:53.765 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:53.834 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:58:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:58:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:53Z" 2022-07-18T03:58:53.861 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:53.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:53Z" } 2022-07-18T03:58:53.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:53.864 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:53.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:53.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:58:58.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:58.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:58:58.868 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:58:58.924 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:58:58.925 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:58:59.004 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:58:59.004 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:58:59.005 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:58:59.005 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:58:59.005 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:58:59.005 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:58:59.006 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:59.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:59.006 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:58:59.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:59.007 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:58:59.007 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:59.007 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:59.007 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:58:59.008 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:59.008 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:58:59.008 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:59.008 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:58:59.009 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:59.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:59.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:59.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:59.010 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:58:59.010 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:59.011 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:59.011 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:58:59.011 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:59.011 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:59.012 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:59.012 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:58:59.012 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:59.013 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:59.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:59.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:59.013 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:58:59.014 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:59.014 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:59.014 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:58:59.014 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:59.015 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:59.015 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:59.015 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:58:59.015 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:59.016 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:59.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:59.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:58:59.016 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:58:59.017 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:58:59.017 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:58:59.017 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:58:59.018 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:58:59.018 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:58:59.018 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:58:59.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:58:59.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:58:59.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:59.020 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:58:59.020 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:58:59.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:59.021 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:58:59.021 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:58:59.021 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:58:59.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:58:59.022 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:58:58Z" 2022-07-18T03:58:59.022 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:58:59.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:58:58Z" } 2022-07-18T03:58:59.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:58:59.025 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:58:59.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:58:59.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:04.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:04.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:04.030 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:04.087 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:04.087 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:04.161 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:04.161 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:04.162 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:04.162 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:04.162 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:04.162 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:04.163 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:04.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:04.163 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:04.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:04.164 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:04.164 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:04.164 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:04.164 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:04.165 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:04.165 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:04.165 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:04.166 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:04.166 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:04.166 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:04.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:04.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:04.167 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:04.167 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:04.167 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:04.168 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:04.168 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:04.168 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:04.169 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:04.169 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:04.169 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:04.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:04.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:04.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:04.170 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:04.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:04.171 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:04.171 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:04.171 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:04.171 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:04.172 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:04.172 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:04.172 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:04.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:04.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:04.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:04.173 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:04.174 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:04.174 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:04.175 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:04.175 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:04.175 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:04.175 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:04.176 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:04.176 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:04.176 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:04.176 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:04.177 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:04.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:04.177 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:04.177 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:04.177 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:04.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:04.178 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:04Z" 2022-07-18T03:59:04.178 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:04.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:04Z" } 2022-07-18T03:59:04.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:04.181 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:04.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:04.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:09.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:09.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:09.185 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:09.235 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:09.235 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:09Z" 2022-07-18T03:59:09.316 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:09.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:09Z" } 2022-07-18T03:59:09.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:09.318 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:09.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:09.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:14.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:14.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:14.322 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:14.378 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:14.379 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:14.457 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:14.457 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:14.457 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:14.458 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:14.458 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:14.458 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:14.460 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:14.460 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:14.460 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:14.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:14.461 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:14.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:14.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:14.461 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:14.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:14.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:14.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:14.462 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:14.463 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:14.463 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:14.463 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:14.463 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:14.464 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:14.464 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:14.464 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:14.464 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:14.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:14.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:14.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:14.465 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:14.466 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:14.466 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:14.466 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:14.466 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:14.467 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:14.467 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:14.467 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:14.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:14.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:14.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:14.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:14.469 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:14.469 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:14.470 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:14.470 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:14.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:14.470 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:14.471 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:14.471 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:14.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:14.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:14.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:14.472 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:14.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:14.472 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:14.473 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:14.473 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:14.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:14.473 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:14Z" 2022-07-18T03:59:14.474 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:14.475 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:14Z" } 2022-07-18T03:59:14.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:14.476 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:14.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:14.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:19.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:19.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:19.481 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:19.530 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:19.531 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:19.605 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:19.606 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:19.606 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:19.606 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:19.607 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:19.607 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:19.607 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:19.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:19.608 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:19.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:19.608 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:19.609 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:19.609 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:19.609 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:19.609 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:19.610 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:19.610 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:19.610 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:19.611 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:19.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:19.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:19.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:19.612 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:19.612 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:19.613 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:19.613 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:19.613 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:19.614 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:19.614 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:19.614 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:19.615 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:19.615 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:19.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:19.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:19.616 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:19.616 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:19.616 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:19.616 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:19.617 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:19.617 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:19.617 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:19.617 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:19.618 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:19.618 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:19.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:19.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:19.619 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:19.619 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:19.620 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:19.620 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:19.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:19.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:19.621 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:19.621 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:19.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:19.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:19.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:19.622 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:19.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:19.623 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:19.623 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:19.623 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:19.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:19.624 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:19Z" 2022-07-18T03:59:19.624 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:19.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:19Z" } 2022-07-18T03:59:19.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:19.626 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:19.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:19.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:24.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:24.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:24.631 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:24.687 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:24.687 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:24.772 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:24.772 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:24.773 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:24.773 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:24.773 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:24.774 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:24.774 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:24.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:24.774 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:24.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:24.775 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:24.775 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:24.775 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:24.776 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:24.776 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:24.777 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:24.777 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:24.777 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:24.778 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:24.778 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:24.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:24.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:24.779 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:24.779 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:24.779 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:24.779 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:24.780 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:24.780 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:24.780 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:24.781 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:24.781 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:24.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:24.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:24.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:24.782 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:24.782 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:24.783 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:24.783 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:24.783 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:24.784 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:24.784 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:24.784 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:24.784 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:24.785 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:24.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:24.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:24.785 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:24.786 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:24.786 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:24.786 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:24.786 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:24.787 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:24.787 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:24.787 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:24.787 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:24.788 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:24.788 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:24.788 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:24.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:24Z" 2022-07-18T03:59:24.790 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:24.791 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:24Z" } 2022-07-18T03:59:24.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:24.793 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:24.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:24.794 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:29.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:29.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:29.797 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:29.853 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:29.854 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:29.920 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:29.921 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:29.921 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:29.922 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:29.922 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:29.922 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:29.923 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:29.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:29.923 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:29.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:29.924 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:29.924 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:29.924 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:29.924 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:29.925 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:29.925 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:29.925 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:29.925 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:29.926 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:29.926 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:29.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:29.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:29.927 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:29.927 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:29.928 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:29.928 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:29.928 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:29.929 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:29.929 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:29.929 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:29.929 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:29.930 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:29.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:29.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:29.931 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:29.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:29.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:29.931 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:29.932 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:29.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:29.932 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:29.932 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:29.933 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:29.933 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:29.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:29.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:29.934 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:29.934 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:29.934 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:29.935 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:29.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:29.935 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:29.935 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:29.936 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:29.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:29.936 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:29.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:29.937 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:29.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:29.938 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:29.938 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:29.938 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:29.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:29.939 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:29Z" 2022-07-18T03:59:29.939 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:29.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:29Z" } 2022-07-18T03:59:29.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:29.941 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:29.942 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:29.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:34.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:34.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:34.947 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:35.003 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:35.004 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:35.085 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:35.085 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:35.086 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:35.086 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:35.086 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:35.086 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:35.087 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:35.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:35.087 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:35.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:35.088 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:35.088 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:35.088 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:35.089 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:35.089 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:35.089 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:35.089 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:35.090 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:35.090 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:35.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:35.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:35.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:35.091 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:35.091 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:35.091 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:35.092 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:35.092 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:35.092 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:35.092 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:35.093 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:35.093 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:35.093 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:35.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:35.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:35.094 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:35.094 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:35.094 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:35.095 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:35.095 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:35.095 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:35.095 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:35.095 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:35.096 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:35.096 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:35.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:35.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:35.097 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:35.097 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:35.097 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:35.097 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:35.098 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:35.103 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:35.104 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:35.104 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:35.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:35.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:35.105 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:35.105 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:35.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:35.105 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:35.106 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:35.106 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:35.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:35.107 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:35Z" 2022-07-18T03:59:35.107 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:35.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:35Z" } 2022-07-18T03:59:35.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:35.110 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:35.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:35.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:40.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:40.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:40.114 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:40.187 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:40.187 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:40.303 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:40.304 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:40.305 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:40.311 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:40.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:40.313 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:40.313 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:40.313 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:40.313 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:40.314 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:40.314 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:40.314 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:40.315 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:40.315 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:40.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:40.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:40.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:40.316 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:40.316 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:40.316 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:40.317 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:40.317 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:40.317 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:40.317 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:40.318 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:40.318 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:40.318 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:40.318 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:40.319 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:40.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:40.319 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:40.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:40.320 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:40.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:40.320 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:40Z" 2022-07-18T03:59:40.320 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:40.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:40Z" } 2022-07-18T03:59:40.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:40.323 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:40.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:40.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:45.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:45.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:45.327 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:45.381 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:45.382 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:45.459 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:45.460 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:45.461 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:45.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:45.462 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:45.463 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:45.464 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:45.464 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:45.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:45.466 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:45.467 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:45.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:45.469 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:45.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:45.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:45.472 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:45.473 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:45.473 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:45.473 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:45.473 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:45.474 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:45.474 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:45.474 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:45.474 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:45.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:45.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:45.475 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:45.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:45.476 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:45.476 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:45.476 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:45.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:45.477 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:45Z" 2022-07-18T03:59:45.477 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:45.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:45Z" } 2022-07-18T03:59:45.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:45.479 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:45.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:45.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:50.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:50.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:50.483 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:50.539 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:50.539 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:50.608 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:50.608 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:50.609 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:50.609 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:50.609 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:50.610 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:50.610 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:50.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:50.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:50.612 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:50.613 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:50.613 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:50.614 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:50.615 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:50.616 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:50.617 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:50.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:50.619 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:50.620 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:50.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:50.622 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:50.623 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:50.624 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:50.626 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:50.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:50.627 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:50Z" 2022-07-18T03:59:50.628 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:50.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:50Z" } 2022-07-18T03:59:50.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:50.632 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:50.633 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:50.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T03:59:55.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:55.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T03:59:55.637 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T03:59:55.685 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T03:59:55.685 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T03:59:55.757 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T03:59:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T03:59:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T03:59:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T03:59:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T03:59:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T03:59:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T03:59:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T03:59:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T03:59:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T03:59:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T03:59:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T03:59:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T03:59:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T03:59:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T03:59:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T03:59:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T03:59:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T03:59:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T03:59:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T03:59:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T03:59:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T03:59:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T03:59:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T03:59:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T03:59:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T03:59:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T03:59:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T03:59:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T03:59:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T03:59:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T03:59:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T03:59:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T03:59:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T03:59:55Z" 2022-07-18T03:59:55.775 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T03:59:55.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T03:59:55Z" } 2022-07-18T03:59:55.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T03:59:55.778 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T03:59:55.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T03:59:55.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:00.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:00.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:00.783 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:00.833 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:00.833 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:00.894 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:00.895 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:00.895 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:00.896 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:00.896 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:00.896 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:00.897 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:00.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:00.897 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:00.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:00.898 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:00.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:00.898 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:00.899 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:00.899 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:00.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:00.899 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:00.900 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:00.900 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:00.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:00.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:00.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:00.901 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:00.901 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:00.901 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:00.902 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:00.902 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:00.902 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:00.902 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:00.903 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:00.903 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:00.903 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:00.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:00.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:00.904 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:00.904 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:00.905 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:00.905 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:00.905 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:00.905 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:00.906 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:00.906 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:00.907 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:00.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:00.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:00.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:00.908 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:00.908 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:00.908 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:00.908 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:00.909 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:00.909 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:00.909 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:00.909 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:00.910 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:00.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:00.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:00.910 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:00.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:00.911 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:00.911 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:00.912 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:00.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:00.912 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:00Z" 2022-07-18T04:00:00.912 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:00.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:00Z" } 2022-07-18T04:00:00.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:00.915 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:00.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:00.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:05.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:05.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:05.920 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:05.973 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:05.973 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:06.048 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:06.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:06.059 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:06.060 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:06.060 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:06Z" 2022-07-18T04:00:06.080 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:06.082 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:06Z" } 2022-07-18T04:00:06.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:06.084 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:06.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:06.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:11.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:11.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:11.090 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:11.145 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:11.146 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:11.219 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:11.219 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:11.219 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:11.219 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:11.220 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:11.220 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:11.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:11.220 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:11.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:11.221 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:11.221 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:11.221 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:11.222 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:11.222 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:11.222 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:11.222 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:11.223 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:11.223 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:11.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:11.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:11.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:11.224 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:11.225 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:11.225 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:11.225 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:11.225 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:11.225 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:11.226 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:11.226 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:11.226 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:11.227 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:11.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:11.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:11.227 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:11.228 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:11.228 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:11.228 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:11.228 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:11.229 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:11.229 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:11.229 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:11.231 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:11.231 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:11.231 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:11.231 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:11.232 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:11.232 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:11.232 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:11.232 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:11.233 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:11.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:11.233 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:11.233 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:11.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:11.234 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:11.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:11.234 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:11.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:11.235 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:11Z" 2022-07-18T04:00:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:11.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:11Z" } 2022-07-18T04:00:11.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:11.238 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:11.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:11.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:16.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:16.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:16.243 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:16.299 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:16.300 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:16.364 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:16.364 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:16.364 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:16.365 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:16.365 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:16.365 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:16.366 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:16.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:16.366 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:16.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:16.367 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:16.367 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:16.367 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:16.368 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:16.368 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:16.368 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:16.369 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:16.369 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:16.369 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:16.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:16.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:16.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:16.370 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:16.370 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:16.371 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:16.371 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:16.371 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:16.371 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:16.372 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:16.372 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:16.372 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:16.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:16.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:16.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:16.373 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:16.374 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:16.374 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:16.374 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:16.374 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:16.375 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:16.375 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:16.375 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:16.375 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:16.376 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:16.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:16.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:16.377 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:16.377 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:16.377 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:16.378 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:16.378 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:16.378 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:16.378 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:16.379 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:16.381 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:16.381 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:16.381 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:16.383 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:16.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:16.383 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:16Z" 2022-07-18T04:00:16.384 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:16.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:16Z" } 2022-07-18T04:00:16.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:16.387 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:16.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:16.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:21.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:21.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:21.392 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:21.448 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:21.448 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:21.511 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:21.511 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:21.512 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:21.512 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:21.512 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:21.513 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:21.513 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:21.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:21.513 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:21.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:21.514 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:21.514 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:21.515 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:21.515 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:21.515 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:21.516 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:21.516 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:21.516 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:21.516 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:21.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:21.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:21.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:21.518 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:21.518 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:21.518 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:21.519 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:21.519 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:21.519 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:21.519 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:21.520 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:21.520 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:21.520 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:21.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:21.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:21.521 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:21.522 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:21.522 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:21.522 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:21.522 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:21.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:21.523 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:21.523 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:21.523 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:21.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:21.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:21.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:21.524 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:21.526 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:21.526 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:21.526 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:21.527 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:21.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:21.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:21.527 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:21.528 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:21.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:21.529 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:21Z" 2022-07-18T04:00:21.529 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:21.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:21Z" } 2022-07-18T04:00:21.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:21.532 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:21.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:21.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:26.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:26.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:26.536 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:26.584 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:26.584 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:26.657 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:26.657 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:26.657 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:26.658 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:26.658 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:26.659 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:26.659 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:26.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:26.659 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:26.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:26.660 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:26.660 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:26.661 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:26.661 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:26.661 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:26.661 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:26.662 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:26.662 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:26.662 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:26.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:26.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:26.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:26.663 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:26.664 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:26.664 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:26.664 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:26.664 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:26.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:26.665 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:26.665 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:26.665 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:26.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:26.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:26.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:26.667 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:26.667 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:26.667 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:26.668 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:26.668 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:26.668 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:26.668 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:26.669 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:26.669 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:26.669 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:26.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:26.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:26.670 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:26.671 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:26.671 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:26.671 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:26.671 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:26.672 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:26.672 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:26.672 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:26.672 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:26.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:26.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:26.673 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:26.674 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:26.675 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:26.676 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:26Z" 2022-07-18T04:00:26.677 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:26.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:26Z" } 2022-07-18T04:00:26.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:26.681 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:26.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:26.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:31.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:31.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:31.685 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:31.733 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:31.734 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:31.790 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:31.790 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:31.791 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:31.791 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:31.791 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:31.791 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:31.792 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:31.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:31.792 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:31.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:31.793 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:31.793 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:31.793 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:31.793 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:31.794 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:31.794 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:31.795 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:31.795 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:31.795 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:31.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:31.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:31.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:31.796 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:31.796 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:31.797 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:31.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:31.799 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:31.800 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:31.801 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:31.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:31.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:31.803 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:31.804 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:31.804 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:31.805 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:31.806 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:31.806 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:31.807 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:31.807 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:31.808 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:31.809 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:31.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:31.811 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:31Z" 2022-07-18T04:00:31.811 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:31.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:31Z" } 2022-07-18T04:00:31.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:31.814 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:31.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:31.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:36.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:36.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:36.817 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:36.872 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:36.873 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:36.944 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:36.945 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:36.945 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:36.945 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:36.946 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:36.946 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:36.946 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:36.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:36.947 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:36.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:36.947 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:36.947 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:36.948 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:36.948 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:36.948 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:36.948 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:36.949 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:36.949 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:36.949 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:36.949 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:36.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:36.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:36.950 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:36.950 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:36.951 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:36.951 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:36.951 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:36.951 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:36.952 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:36.952 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:36.952 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:36.952 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:36.953 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:36.954 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:36.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:36.956 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:36.957 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:36.958 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:36.959 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:36.960 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:36Z" 2022-07-18T04:00:36.961 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:36.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:36Z" } 2022-07-18T04:00:36.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:36.963 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:36.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:36.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:41.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:41.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:41.967 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:42.023 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:42.023 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:42.088 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:42.088 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:42.089 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:42.089 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:42.089 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:42.090 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:42.090 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:42.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:42.091 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:42.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:42.092 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:42.092 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:42.092 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:42.093 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:42.093 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:42.093 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:42.093 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:42.094 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:42.094 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:42.094 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:42.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:42.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:42.095 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:42.095 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:42.095 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:42.096 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:42.096 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:42.096 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:42.097 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:42.097 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:42.097 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:42.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:42.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:42.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:42.098 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:42.099 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:42.099 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:42.099 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:42.100 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:42.100 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:42.100 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:42.101 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:42.101 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:42.101 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:42.103 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:42.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:42.105 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:42.106 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:42Z" 2022-07-18T04:00:42.107 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:42.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:42Z" } 2022-07-18T04:00:42.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:42.109 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:42.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:42.110 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:47.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:47.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:47.113 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:47.168 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:47.169 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:47.245 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:47.246 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:47.246 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:47.246 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:47.247 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:47.247 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:47.247 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:47.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:47.248 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:47.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:47.248 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:47.248 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:47.249 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:47.249 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:47.249 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:47.250 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:47.250 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:47.250 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:47.251 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:47.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:47.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:47.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:47.252 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:47.252 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:47.252 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:47.252 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:47.252 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:47.253 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:47.253 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:47.253 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:47.253 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:47.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:47.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:47.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:47.255 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:47.256 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:47.257 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:47.258 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:47.258 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:47.258 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:47.259 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:47.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:47.261 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:47.262 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:47Z" 2022-07-18T04:00:47.262 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:47.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:47Z" } 2022-07-18T04:00:47.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:47.265 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:47.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:47.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:52.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:52.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:52.269 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:52.319 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:52.319 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:52.382 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:52.383 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:52.383 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:52.383 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:52.383 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:52.384 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:52.384 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:52.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:52.384 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:52.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:52.385 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:52.385 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:52.386 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:52.386 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:52.386 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:52.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:52.387 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:52.387 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:52.387 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:52.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:52.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:52.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:52.389 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:52.389 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:52.389 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:52.389 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:52.390 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:52.390 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:52.390 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:52.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:52.391 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:52.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:52.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:52.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:52.392 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:52.392 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:52.392 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:52.392 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:52.393 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:52.393 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:52.393 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:52.393 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:52.394 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:52.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:52.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:52.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:52.395 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:52.395 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:52.395 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:52.396 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:52.396 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:52.396 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:52.397 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:52.397 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:52.397 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:52.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:52.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:52.399 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:52.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:52.400 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:52.400 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:52.401 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:52.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:52.401 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:52Z" 2022-07-18T04:00:52.402 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:52.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:52Z" } 2022-07-18T04:00:52.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:52.406 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:52.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:52.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:00:57.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:57.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:00:57.410 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:00:57.467 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:00:57.467 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:00:57.542 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:00:57.543 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:00:57.543 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:00:57.544 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:00:57.544 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:00:57.544 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:00:57.544 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:57.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:57.545 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:00:57.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:57.545 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:00:57.546 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:57.546 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:57.546 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:00:57.547 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:57.547 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:00:57.547 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:57.547 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:00:57.548 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:57.548 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:57.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:57.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:57.549 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:00:57.549 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:57.549 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:57.549 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:00:57.550 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:57.550 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:57.550 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:57.551 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:00:57.551 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:57.551 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:57.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:57.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:57.552 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:00:57.552 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:57.553 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:57.553 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:00:57.553 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:57.553 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:57.554 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:57.554 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:00:57.554 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:57.555 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:57.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:57.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:00:57.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:00:57.556 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:00:57.556 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:00:57.556 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:00:57.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:00:57.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:00:57.557 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:00:57.557 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:00:57.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:00:57.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:57.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:00:57.558 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:00:57.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:57.559 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:00:57.559 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:00:57.559 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:00:57.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:00:57.560 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:00:57Z" 2022-07-18T04:00:57.560 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:00:57.561 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:00:57Z" } 2022-07-18T04:00:57.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:00:57.563 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:00:57.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:00:57.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:02.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:02.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:02.567 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:02.622 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:02.623 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:02.691 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:02.691 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:02.692 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:02.692 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:02.693 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:02.694 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:02.694 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:02.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:02.695 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:02.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:02.696 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:02.696 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:02.697 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:02.697 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:02.698 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:02.698 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:02.699 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:02.699 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:02.700 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:02.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:02.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:02.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:02.702 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:02.703 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:02.703 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:02.704 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:02.704 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:02.705 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:02.705 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:02.706 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:02.706 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:02.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:02.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:02.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:02.708 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:02.708 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:02.709 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:02.709 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:02.710 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:02.710 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:02.710 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:02.710 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:02.712 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:02.712 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:02.712 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:02.713 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:02.713 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:02.713 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:02.713 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:02.714 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:02.714 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:02.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:02.715 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:02.716 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:02Z" 2022-07-18T04:01:02.717 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:02.718 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:02Z" } 2022-07-18T04:01:02.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:02.719 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:02.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:02.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:07.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:07.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:07.723 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:07.779 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:07.780 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:07.859 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:07.860 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:07.861 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:07.861 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:07.861 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:07.862 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:07.862 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:07.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:07.864 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:07.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:07.865 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:07.865 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:07.866 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:07.866 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:07.867 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:07.867 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:07.868 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:07.868 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:07.869 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:07.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:07.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:07.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:07.871 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:07.871 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:07.872 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:07.872 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:07.873 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:07.873 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:07.874 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:07.874 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:07.874 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:07.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:07.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:07.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:07.877 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:07.877 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:07.878 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:07.878 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:07.879 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:07.879 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:07.880 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:07.880 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:07.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:07.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:07.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:07.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:07.883 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:07.883 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:07.884 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:07.885 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:07.886 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:07.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:07.888 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:07.889 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:07.890 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:07.891 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:07Z" 2022-07-18T04:01:07.892 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:07.893 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:07Z" } 2022-07-18T04:01:07.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:07.895 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:07.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:07.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:12.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:12.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:12.900 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:12.958 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:12.958 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:13.021 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:13.021 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:13.022 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:13.022 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:13.022 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:13.023 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:13.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:13.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:13.026 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:13.026 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:13.026 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:13.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:13.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:13.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:13.027 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:13.028 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:13.028 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:13.028 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:13.028 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:13.029 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:13.029 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:13.029 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:13.029 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:13.030 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:13.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:13.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:13.031 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:13.031 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:13.031 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:13.031 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:13.032 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:13.032 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:13.032 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:13.032 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:13.033 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:13.033 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:13.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:13.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:13.034 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:13.035 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:13.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:13.037 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:13.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:13.039 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:13Z" 2022-07-18T04:01:13.039 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:13.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:13Z" } 2022-07-18T04:01:13.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:13.041 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:13.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:13.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:18.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:18.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:18.046 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:18.097 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:18.098 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:18.157 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:18.158 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:18.158 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:18.158 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:18.159 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:18.159 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:18.159 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:18.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:18.160 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:18.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:18.160 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:18.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:18.161 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:18.161 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:18.161 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:18.161 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:18.162 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:18.162 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:18.162 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:18.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:18.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:18.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:18.163 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:18.164 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:18.164 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:18.165 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:18.165 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:18.165 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:18.165 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:18.166 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:18.166 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:18.166 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:18.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:18.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:18.167 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:18.167 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:18.167 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:18.168 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:18.168 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:18.168 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:18.168 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:18.169 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:18.169 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:18.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:18.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:18.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:18.170 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:18.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:18.170 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:18.171 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:18.171 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:18.171 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:18.171 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:18.172 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:18.172 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:18.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:18.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:18.173 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:18.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:18.173 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:18.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:18.174 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:18.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:18.175 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:18Z" 2022-07-18T04:01:18.175 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:18.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:18Z" } 2022-07-18T04:01:18.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:18.178 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:18.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:18.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:23.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:23.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:23.182 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:23.237 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:23.237 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:23.301 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:23.301 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:23.301 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:23.302 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:23.302 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:23.302 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:23.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:23.307 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:23.308 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:23.309 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:23.309 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:23.309 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:23.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:23.310 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:23.311 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:23.312 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:23.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:23.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:23.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:23.314 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:23.315 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:23.315 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:23.317 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:23.318 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:23.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:23.319 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:23.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:23.319 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:23Z" 2022-07-18T04:01:23.320 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:23.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:23Z" } 2022-07-18T04:01:23.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:23.322 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:23.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:23.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:28.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:28.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:28.327 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:28.383 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:28.383 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:28.456 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:28.457 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:28.457 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:28.458 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:28.459 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:28.460 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:28.461 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:28.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:28.463 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:28.464 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:28.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:28.466 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:28.467 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:28.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:28.469 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:28.470 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:28.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:28.472 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:28.473 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:28Z" 2022-07-18T04:01:28.473 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:28.475 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:28Z" } 2022-07-18T04:01:28.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:28.476 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:28.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:28.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:33.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:33.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:33.481 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:33.536 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:33.537 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:33.619 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:33Z" 2022-07-18T04:01:33.637 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:33.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:33Z" } 2022-07-18T04:01:33.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:33.640 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:33.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:33.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:38.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:38.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:38.643 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:38.692 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:38.693 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:38.752 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:38.753 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:38.753 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:38.753 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:38.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:38.755 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:38.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:38.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:38.758 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:38.759 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:38.760 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:38.761 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:38.762 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:38.763 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:38.763 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:38.763 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:38.763 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:38.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:38.765 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:38.766 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:38.766 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:38.766 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:38.766 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:38.767 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:38.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:38.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:38.767 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:38.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:38.768 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:38.768 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:38.769 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:38.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:38.769 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:38Z" 2022-07-18T04:01:38.769 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:38.771 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:38Z" } 2022-07-18T04:01:38.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:38.772 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:38.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:38.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:43.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:43.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:43.775 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:43.830 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:43.830 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:43.917 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:43.918 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:43.919 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:43.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:43.921 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:43.922 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:43.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:43.924 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:43.925 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:43.926 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:43.927 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:43.928 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:43.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:43.930 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:43.931 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:43.932 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:43.933 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:43.933 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:43.933 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:43.934 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:43.934 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:43.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:43.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:43.936 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:43Z" 2022-07-18T04:01:43.936 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:43.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:43Z" } 2022-07-18T04:01:43.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:43.939 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:43.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:43.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:48.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:48.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:48.943 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:49.000 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:49.001 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:49.071 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:49.071 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:49.072 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:49.074 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:49.075 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:49.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:49.077 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:49.078 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:49.079 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:49.079 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:49.079 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:49.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:49.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:49.080 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:49.080 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:49.080 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:49.081 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:49.081 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:49.081 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:49.081 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:49.082 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:49.082 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:49.082 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:49.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:49.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:49.083 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:49.084 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:49.085 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:49.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:49.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:49.088 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:49Z" 2022-07-18T04:01:49.088 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:49.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:49Z" } 2022-07-18T04:01:49.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:49.090 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:49.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:49.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:54.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:54.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:54.095 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:54.150 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:54.151 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:54.230 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:54.230 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:54.230 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:54.231 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:54.231 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:54.231 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:54.232 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:54.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:54.232 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:54.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:54.233 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:54.233 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:54.233 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:54.233 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:54.234 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:54.234 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:54.234 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:54.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:54.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:54.236 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:54.236 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:54.236 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:54.237 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:54.237 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:54.237 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:54.237 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:54.238 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:54.238 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:54.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:54.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:54.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:54.239 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:54.239 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:54.240 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:54.241 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:54.242 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:54.243 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:54.243 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:54.243 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:54.243 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:54.244 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:54.244 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:54.244 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:54.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:54.245 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:54.245 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:54.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:54.245 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:54.246 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:54.246 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:54.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:54.246 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:54Z" 2022-07-18T04:01:54.247 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:54.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:54Z" } 2022-07-18T04:01:54.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:54.249 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:54.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:54.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:01:59.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:59.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:01:59.253 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:01:59.310 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:01:59.311 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:01:59.402 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:01:59.402 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:01:59.402 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:01:59.403 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:01:59.403 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:01:59.403 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:01:59.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:59.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:59.404 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:01:59.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:59.405 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:01:59.405 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:59.405 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:59.406 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:01:59.406 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:59.406 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:01:59.406 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:59.407 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:01:59.407 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:59.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:59.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:59.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:59.408 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:01:59.408 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:59.408 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:01:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:59.409 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:59.410 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:01:59.410 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:59.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:59.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:59.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:59.412 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:01:59.413 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:59.413 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:59.414 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:01:59.414 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:59.415 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:59.415 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:59.415 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:01:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:59.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:01:59.417 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:01:59.418 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:01:59.418 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:01:59.418 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:01:59.419 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:01:59.420 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:01:59.421 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:01:59.421 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:01:59.422 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:01:59.422 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:59.423 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:01:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:01:59.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:01:59.426 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:01:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:01:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:01:59.428 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:01:59Z" 2022-07-18T04:01:59.428 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:01:59.431 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:01:59Z" } 2022-07-18T04:01:59.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:01:59.432 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:01:59.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:01:59.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:04.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:04.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:04.437 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:04.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:04.491 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:04.581 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:04.581 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:04.582 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:04.583 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:04.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:04.583 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:04.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:04.584 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:04.584 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:04.584 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:04.584 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:04.585 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:04.585 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:04.585 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:04.585 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:04.586 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:04.586 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:04.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:04.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:04.587 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:04.587 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:04.587 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:04.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:04.588 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:04.588 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:04.589 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:04.589 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:04.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:04.589 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:04.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:04.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:04.590 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:04.590 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:04.591 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:04.591 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:04.591 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:04.591 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:04.592 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:04.592 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:04.592 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:04.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:04.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:04.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:04.593 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:04.593 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:04.594 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:04.594 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:04.594 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:04.595 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:04.595 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:04.595 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:04.595 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:04.596 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:04.596 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:04.596 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:04.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:04.597 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:04.597 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:04.597 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:04.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:04.598 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:04Z" 2022-07-18T04:02:04.598 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:04.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:04Z" } 2022-07-18T04:02:04.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:04.601 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:04.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:04.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:09.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:09.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:09.605 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:09.661 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:09.661 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:09.723 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:09.723 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:09.724 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:09.724 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:09.724 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:09.724 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:09.725 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:09.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:09.725 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:09.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:09.726 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:09.726 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:09.726 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:09.727 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:09.727 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:09.727 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:09.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:09.728 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:09.728 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:09.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:09.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:09.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:09.729 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:09.729 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:09.729 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:09.730 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:09.730 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:09.730 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:09.730 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:09.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:09.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:09.732 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:09.732 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:09.733 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:09.734 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:09.734 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:09.734 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:09.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:09.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:09.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:09.735 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:09.735 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:09.736 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:09.737 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:09.737 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:09.737 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:09.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:09.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:09.738 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:09.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:09.739 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:09.739 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:09.739 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:09.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:09.740 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:09Z" 2022-07-18T04:02:09.741 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:09.743 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:09Z" } 2022-07-18T04:02:09.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:09.744 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:09.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:09.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:14.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:14.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:14.748 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:14.804 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:14.804 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:14.889 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:14.889 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:14.889 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:14.890 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:14.890 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:14.890 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:14.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:14.891 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:14.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:14.891 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:14.891 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:14.892 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:14.892 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:14.892 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:14.892 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:14.893 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:14.893 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:14.893 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:14.893 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:14.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:14.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:14.894 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:14.895 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:14.895 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:14.895 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:14.895 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:14.895 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:14.896 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:14.896 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:14.896 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:14.897 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:14.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:14.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:14.897 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:14.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:14.898 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:14.898 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:14.898 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:14.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:14.899 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:14.899 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:14.899 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:14.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:14.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:14.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:14.900 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:14.901 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:14.901 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:14.901 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:14.901 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:14.902 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:14.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:14.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:14.905 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:14Z" 2022-07-18T04:02:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:14.906 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:14Z" } 2022-07-18T04:02:14.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:14.907 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:14.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:14.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:19.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:19.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:19.911 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:19.967 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:19.967 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:20.030 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:20.030 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:20.031 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:20.031 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:20.031 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:20.031 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:20.032 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:20.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:20.032 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:20.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:20.033 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:20.033 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:20.033 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:20.033 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:20.034 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:20.034 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:20.034 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:20.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:20.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:20.036 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:20.036 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:20.037 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:20.038 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:20.039 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:20.040 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:20.041 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:20.041 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:20.041 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:20.041 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:20.042 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:20.043 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:20.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:20.043 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:20.043 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:20.044 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:20.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:20.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:20.044 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:20.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:20.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:20.045 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:20.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:20.046 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:20.046 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:20.046 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:20.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:20.047 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:20Z" 2022-07-18T04:02:20.047 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:20.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:20Z" } 2022-07-18T04:02:20.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:20.050 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:20.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:20.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:25.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:25.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:25.054 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:25.109 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:25.110 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:25.191 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:25.191 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:25.191 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:25.192 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:25.192 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:25.192 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:25.192 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:25.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:25.193 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:25.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:25.193 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:25.194 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:25.194 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:25.194 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:25.195 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:25.195 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:25.195 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:25.195 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:25.196 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:25.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:25.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:25.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:25.198 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:25.198 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:25.198 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:25.198 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:25.199 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:25.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:25.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:25.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:25.200 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:25.200 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:25.200 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:25.200 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:25.201 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:25.201 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:25.201 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:25.201 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:25.202 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:25.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:25.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:25.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:25.203 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:25.203 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:25.203 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:25.203 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:25.204 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:25.204 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:25.204 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:25.204 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:25.205 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:25.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:25.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:25.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:25.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:25.206 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:25.206 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:25.207 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:25.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:25.207 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:25Z" 2022-07-18T04:02:25.207 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:25.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:25Z" } 2022-07-18T04:02:25.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:25.210 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:25.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:25.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:30.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:30.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:30.214 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:30.262 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:30.263 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:30.322 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:30.323 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:30.323 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:30.324 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:30.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:30.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:30.326 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:30.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:30.327 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:30.327 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:30.328 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:30.328 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:30.328 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:30.328 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:30.328 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:30.329 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:30.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:30.331 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:30.331 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:30.331 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:30.331 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:30.332 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:30.332 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:30.332 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:30.332 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:30.333 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:30.333 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:30.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:30.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:30.334 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:30.334 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:30.335 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:30.335 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:30.335 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:30.335 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:30.336 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:30.336 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:30.336 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:30.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:30.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:30.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:30.338 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:30.339 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:30.339 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:30.339 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:30.340 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:30.340 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:30.340 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:30.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:30.341 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:30.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:30.341 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:30.341 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:30.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:30.342 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:30.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:30.342 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:30.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:30.343 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:30Z" 2022-07-18T04:02:30.343 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:30.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:30Z" } 2022-07-18T04:02:30.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:30.346 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:30.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:30.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:35.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:35.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:35.350 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:35.402 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:35.402 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:35.552 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:35.553 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:35.553 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:35.556 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:35.557 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:35.557 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:35.557 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:35.557 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:35.558 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:35.559 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:35.560 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:35.561 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:35.562 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:35.563 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:35.564 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:35.565 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:35.565 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:35.565 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:35.565 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:35.566 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:35.567 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:35.568 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:35Z" 2022-07-18T04:02:35.569 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:35.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:35Z" } 2022-07-18T04:02:35.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:35.572 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:35.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:35.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:40.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:40.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:40.575 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:40.631 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:40.632 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:40.697 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:40.698 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:40.698 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:40.699 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:40.699 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:40.699 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:40.700 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:40.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:40.700 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:40.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:40.701 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:40.702 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:40.702 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:40.702 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:40.703 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:40.703 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:40.703 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:40.703 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:40.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:40.705 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:40.706 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:40.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:40.708 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:40.709 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:40.709 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:40.709 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:40.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:40.711 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:40.712 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:40.713 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:40.714 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:40Z" 2022-07-18T04:02:40.715 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:40.716 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:40Z" } 2022-07-18T04:02:40.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:40.718 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:40.718 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:40.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:45.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:45.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:45.721 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:45.779 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:45.779 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:45.945 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:45.945 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:45.945 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:45.946 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:45.946 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:45.946 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:45.947 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:45.948 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:45.949 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:45.950 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:45.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:45.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:45.951 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:45.951 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:45.952 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:45.953 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:45.954 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:45.955 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:45.956 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:45.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:45.958 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:45.959 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:45.960 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:45.961 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:45.962 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:45Z" 2022-07-18T04:02:45.963 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:45.964 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:45Z" } 2022-07-18T04:02:45.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:45.965 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:45.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:45.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:50.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:50.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:50.969 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:51.026 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:51.026 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:51.108 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:51.109 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:51.109 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:51.110 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:51.110 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:51.111 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:51.111 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:51.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:51.113 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:51.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:51.114 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:51.114 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:51.115 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:51.115 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:51.115 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:51.116 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:51.116 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:51.117 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:51.117 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:51.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:51.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:51.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:51.119 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:51.119 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:51.120 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:51.120 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:51.121 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:51.121 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:51.122 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:51.122 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:51.123 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:51.123 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:51.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:51.124 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:51.126 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:51.126 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:51.126 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:51.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:51.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:51.128 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:51.128 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:51.129 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:51.129 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:51.130 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:51.130 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:51.131 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:51.131 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:51.131 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:51.132 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:51.132 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:51.133 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:51.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:51.134 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:51.134 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:51.135 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:51.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:51.136 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:51Z" 2022-07-18T04:02:51.136 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:51.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:51Z" } 2022-07-18T04:02:51.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:51.141 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:51.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:51.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:02:56.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:56.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:02:56.146 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:02:56.203 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:02:56.204 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:02:56.268 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:02:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:02:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:02:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:02:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:02:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:02:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:02:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:02:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:02:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:02:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:02:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:02:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:02:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:02:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:02:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:02:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:02:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:02:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:02:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:02:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:02:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:02:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:02:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:02:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:02:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:02:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:02:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:02:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:02:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:02:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:02:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:02:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:02:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:02:56Z" 2022-07-18T04:02:56.286 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:02:56.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:02:56Z" } 2022-07-18T04:02:56.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:02:56.288 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:02:56.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:02:56.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:01.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:01.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:01.294 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:01.351 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:01.352 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:01.436 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:01.436 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:01.436 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:01.437 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:01.437 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:01.437 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:01.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:01.438 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:01.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:01.438 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:01.438 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:01.439 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:01.439 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:01.439 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:01.440 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:01.440 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:01.440 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:01.440 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:01.441 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:01.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:01.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:01.441 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:01.442 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:01.442 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:01.442 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:01.443 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:01.443 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:01.443 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:01.443 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:01.444 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:01.444 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:01.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:01.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:01.445 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:01.445 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:01.445 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:01.446 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:01.446 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:01.446 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:01.446 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:01.447 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:01.447 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:01.447 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:01.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:01.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:01.448 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:01.448 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:01.448 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:01.449 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:01.449 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:01.449 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:01.450 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:01.450 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:01.450 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:01.450 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:01.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:01.451 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:01.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:01.451 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:01.452 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:01.452 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:01.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:01.452 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:01Z" 2022-07-18T04:03:01.453 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:01.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:01Z" } 2022-07-18T04:03:01.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:01.455 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:01.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:01.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:06.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:06.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:06.460 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:06.515 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:06.516 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:06.577 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:06.578 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:06.578 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:06.578 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:06.579 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:06.579 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:06.579 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:06.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:06.580 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:06.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:06.580 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:06.580 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:06.582 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:06.582 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:06.582 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:06.583 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:06.583 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:06.583 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:06.583 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:06.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:06.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:06.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:06.584 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:06.585 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:06.585 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:06.585 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:06.585 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:06.586 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:06.586 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:06.586 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:06.586 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:06.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:06.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:06.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:06.587 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:06.588 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:06.588 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:06.588 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:06.588 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:06.589 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:06.589 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:06.589 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:06.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:06.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:06.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:06.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:06.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:06.591 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:06.591 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:06.591 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:06.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:06.592 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:06.592 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:06.592 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:06.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:06.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:06.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:06.593 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:06.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:06.594 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:06.594 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:06.595 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:06.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:06.595 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:06Z" 2022-07-18T04:03:06.595 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:06.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:06Z" } 2022-07-18T04:03:06.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:06.598 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:06.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:06.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:11.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:11.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:11.602 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:11.653 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:11.653 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:11.732 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:11.733 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:11.733 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:11.734 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:11.734 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:11.735 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:11.735 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:11.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:11.736 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:11.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:11.737 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:11.738 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:11.738 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:11.738 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:11.738 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:11.739 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:11.739 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:11.739 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:11.740 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:11.740 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:11.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:11.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:11.741 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:11.741 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:11.741 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:11.742 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:11.742 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:11.742 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:11.742 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:11.743 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:11.743 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:11.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:11.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:11.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:11.744 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:11.744 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:11.745 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:11.745 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:11.745 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:11.745 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:11.746 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:11.746 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:11.746 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:11.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:11.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:11.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:11.747 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:11.748 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:11.748 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:11.748 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:11.748 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:11.749 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:11.749 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:11.749 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:11.749 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:11.750 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:11.750 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:11.750 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:11.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:11.750 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:11.751 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:11.751 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:11.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:11.751 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:11Z" 2022-07-18T04:03:11.752 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:11.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:11Z" } 2022-07-18T04:03:11.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:11.755 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:11.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:11.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:16.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:16.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:16.760 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:16.815 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:16.816 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:16.900 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:16.901 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:16.901 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:16.902 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:16.902 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:16.903 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:16.903 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:16.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:16.904 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:16.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:16.904 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:16.905 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:16.905 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:16.905 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:16.905 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:16.906 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:16.906 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:16.906 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:16.907 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:16.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:16.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:16.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:16.908 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:16.908 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:16.908 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:16.909 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:16.909 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:16.909 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:16.909 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:16.909 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:16.910 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:16.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:16.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:16.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:16.911 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:16.911 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:16.911 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:16.911 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:16.912 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:16.912 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:16.912 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:16.913 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:16.913 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:16.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:16.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:16.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:16.914 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:16.914 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:16.914 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:16.915 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:16.915 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:16.915 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:16.915 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:16.916 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:16.916 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:16.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:16.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:16.917 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:16.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:16.917 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:16.917 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:16.918 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:16.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:16.918 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:16Z" 2022-07-18T04:03:16.919 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:16.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:16Z" } 2022-07-18T04:03:16.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:16.921 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:16.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:16.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:21.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:21.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:21.926 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:21.978 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:21.979 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:22.038 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:22.039 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:22.040 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:22.040 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:22.041 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:22.041 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:22.042 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:22.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:22.043 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:22.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:22.044 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:22.045 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:22.045 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:22.046 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:22.046 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:22.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:22.047 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:22.047 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:22.048 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:22.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:22.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:22.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:22.049 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:22.049 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:22.049 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:22.049 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:22.050 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:22.050 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:22.050 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:22.050 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:22.051 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:22.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:22.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:22.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:22.052 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:22.053 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:22.053 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:22.053 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:22.054 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:22.054 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:22.054 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:22.055 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:22.055 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:22.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:22.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:22.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:22.056 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:22.056 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:22.057 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:22.057 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:22.057 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:22.057 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:22.058 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:22.058 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:22.058 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:22.058 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:22.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:22.059 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:22.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:22.060 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:22.060 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:22.060 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:22.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:22.061 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:22Z" 2022-07-18T04:03:22.061 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:22.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:22Z" } 2022-07-18T04:03:22.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:22.064 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:22.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:22.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:27.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:27.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:27.070 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:27.124 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:27.124 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:27.186 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:27.186 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:27.186 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:27.187 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:27.187 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:27.187 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:27.187 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:27.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:27.188 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:27.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:27.188 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:27.189 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:27.189 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:27.189 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:27.190 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:27.190 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:27.190 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:27.190 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:27.191 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:27.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:27.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:27.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:27.192 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:27.192 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:27.192 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:27.193 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:27.193 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:27.193 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:27.194 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:27.194 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:27.194 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:27.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:27.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:27.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:27.195 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:27.195 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:27.196 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:27.196 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:27.196 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:27.196 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:27.197 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:27.197 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:27.197 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:27.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:27.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:27.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:27.198 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:27.199 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:27.199 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:27.199 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:27.199 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:27.200 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:27.200 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:27.200 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:27.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:27.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:27.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:27.201 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:27.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:27.202 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:27.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:27.202 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:27.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:27.203 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:27Z" 2022-07-18T04:03:27.203 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:27.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:27Z" } 2022-07-18T04:03:27.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:27.206 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:27.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:27.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:32.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:32.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:32.210 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:32.264 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:32.264 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:32.343 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:32.344 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:32.344 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:32.344 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:32.344 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:32.345 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:32.345 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:32.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:32.345 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:32.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:32.346 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:32.346 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:32.347 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:32.347 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:32.347 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:32.348 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:32.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:32.350 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:32.350 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:32.350 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:32.350 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:32.351 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:32.352 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:32.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:32.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:32.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:32.353 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:32.353 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:32.353 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:32.353 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:32.354 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:32.354 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:32.354 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:32.354 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:32.355 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:32.355 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:32.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:32.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:32.356 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:32.356 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:32.356 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:32.356 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:32.357 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:32.357 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:32.357 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:32.357 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:32.358 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:32.358 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:32.358 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:32.359 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:32.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:32.359 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:32.360 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:32.360 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:32.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:32.360 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:32Z" 2022-07-18T04:03:32.361 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:32.362 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:32Z" } 2022-07-18T04:03:32.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:32.363 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:32.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:32.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:37.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:37.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:03:37.367 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:03:37.422 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:37.423 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:37.499 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:03:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:03:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:03:37.500 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:03:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:37.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:37.502 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:03:37.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:37.503 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:03:37.503 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:37.504 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:37.504 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:03:37.505 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:37.505 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:03:37.506 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:37.506 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:03:37.507 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:37.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:37.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:37.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:37.509 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:03:37.509 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:37.510 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:37.510 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:03:37.511 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:37.511 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:37.512 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:37.512 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:03:37.513 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:37.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:37.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:37.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:37.515 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:03:37.515 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:37.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:37.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:03:37.517 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:37.517 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:37.518 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:37.518 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:03:37.519 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:37.519 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:37.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:37.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:37.521 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:03:37.521 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:37.522 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:03:37.522 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:03:37.522 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:03:37.523 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:03:37.523 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:03:37.524 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:03:37.524 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:37.525 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:37.525 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:37.526 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:37.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:37.527 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:03:37.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:03:37.528 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:03:37.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:37.529 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:03:37Z" 2022-07-18T04:03:37.529 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:37.531 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:03:37Z" } 2022-07-18T04:03:37.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:03:37.533 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:03:37.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:03:37.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:03:42.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2022-07-18T04:03:42.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T04:03:42.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T04:03:42.549 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T04:03:42.603 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:42.604 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:42.628 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T04:03:42.628+0000 7f8447394540 0 failed reading obj info from .rgw.root:zone_info.9b07f48a-ef73-4ad5-94be-e184c9a423d4: (2) No such file or directory 2022-07-18T04:03:42.628 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T04:03:42.628+0000 7f8447394540 0 WARNING: could not read zone params for zone id=9b07f48a-ef73-4ad5-94be-e184c9a423d4 name=test-zone1 2022-07-18T04:03:42.629 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T04:03:42.629+0000 7f8447394540 0 failed reading obj info from .rgw.root:zone_info.ea63e6cf-483a-453e-81a7-bf1983b4a211: (2) No such file or directory 2022-07-18T04:03:42.630 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-18T04:03:42.629+0000 7f8447394540 0 WARNING: could not read zone params for zone id=ea63e6cf-483a-453e-81a7-bf1983b4a211 name=test-zone3 2022-07-18T04:03:42.634 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:42.635 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T04:03:42.635 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone2", 2022-07-18T04:03:42.635 INFO:teuthology.orchestra.run.smithi159.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-07-18T04:03:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "control_pool": "test-zone2.rgw.control", 2022-07-18T04:03:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-07-18T04:03:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-07-18T04:03:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "log_pool": "test-zone2.rgw.log", 2022-07-18T04:03:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-07-18T04:03:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-07-18T04:03:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-07-18T04:03:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-07-18T04:03:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-07-18T04:03:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-07-18T04:03:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-07-18T04:03:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-07-18T04:03:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-07-18T04:03:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "system_key": { 2022-07-18T04:03:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "access_key": "6VJB8SA94IKXZOZ9", 2022-07-18T04:03:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "secret_key": "HQA6fC6al2pexDLZUO1RbSRo8VNNsqIg" 2022-07-18T04:03:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_pools": [ 2022-07-18T04:03:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "default-placement", 2022-07-18T04:03:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:03:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-07-18T04:03:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": { 2022-07-18T04:03:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD": { 2022-07-18T04:03:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-07-18T04:03:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "compression_type": "random" 2022-07-18T04:03:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-07-18T04:03:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "index_type": 0 2022-07-18T04:03:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T04:03:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-07-18T04:03:42.651 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:42.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T04:03:42.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T04:03:42.655 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-07-18T04:03:42.705 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:03:42.705 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:03:42.757 INFO:teuthology.orchestra.run.smithi159.stderr:Sending period to new master zone 9b07f48a-ef73-4ad5-94be-e184c9a423d4 2022-07-18T04:03:42.802 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:03:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "epoch": 5, 2022-07-18T04:03:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "predecessor_uuid": "af1b1dd9-a63b-4529-89f0-3f46b7648654", 2022-07-18T04:03:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": [], 2022-07-18T04:03:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "period_map": { 2022-07-18T04:03:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:03:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "zonegroups": [ 2022-07-18T04:03:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "test-zonegroup", 2022-07-18T04:03:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zonegroup", 2022-07-18T04:03:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "api_name": "", 2022-07-18T04:03:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "is_master": true, 2022-07-18T04:03:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T04:03:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T04:03:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "hostnames": [], 2022-07-18T04:03:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "hostnames_s3website": [], 2022-07-18T04:03:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T04:03:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "zones": [ 2022-07-18T04:03:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T04:03:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone1", 2022-07-18T04:03:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T04:03:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8000" 2022-07-18T04:03:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T04:03:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T04:03:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T04:03:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T04:03:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T04:03:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T04:03:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T04:03:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T04:03:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T04:03:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T04:03:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "a24354e3-daf6-4450-84a1-d728d5c57994", 2022-07-18T04:03:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone4", 2022-07-18T04:03:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T04:03:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi159.front.sepia.ceph.com:8001" 2022-07-18T04:03:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T04:03:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T04:03:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T04:03:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T04:03:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "pubsub", 2022-07-18T04:03:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": false, 2022-07-18T04:03:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [ 2022-07-18T04:03:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "test-zone1" 2022-07-18T04:03:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T04:03:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T04:03:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T04:03:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T04:03:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone2", 2022-07-18T04:03:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T04:03:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi159.front.sepia.ceph.com:8000" 2022-07-18T04:03:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T04:03:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T04:03:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T04:03:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T04:03:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T04:03:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": false, 2022-07-18T04:03:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T04:03:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T04:03:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T04:03:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T04:03:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "id": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T04:03:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "test-zone3", 2022-07-18T04:03:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "endpoints": [ 2022-07-18T04:03:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "http://smithi133.front.sepia.ceph.com:8001" 2022-07-18T04:03:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "log_meta": false, 2022-07-18T04:03:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "log_data": true, 2022-07-18T04:03:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_index_max_shards": 11, 2022-07-18T04:03:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "read_only": false, 2022-07-18T04:03:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "tier_type": "", 2022-07-18T04:03:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from_all": true, 2022-07-18T04:03:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_from": [], 2022-07-18T04:03:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "redirect_zone": "", 2022-07-18T04:03:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "supported_features": [ 2022-07-18T04:03:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T04:03:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "placement_targets": [ 2022-07-18T04:03:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "name": "default-placement", 2022-07-18T04:03:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "tags": [], 2022-07-18T04:03:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "storage_classes": [ 2022-07-18T04:03:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "STANDARD" 2022-07-18T04:03:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "default_placement": "default-placement", 2022-07-18T04:03:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T04:03:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_policy": { 2022-07-18T04:03:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: "groups": [] 2022-07-18T04:03:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled_features": [ 2022-07-18T04:03:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: "resharding" 2022-07-18T04:03:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2022-07-18T04:03:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: "short_zone_ids": [ 2022-07-18T04:03:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.833 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T04:03:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 3166817753 2022-07-18T04:03:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.835 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "a24354e3-daf6-4450-84a1-d728d5c57994", 2022-07-18T04:03:42.835 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 2779073914 2022-07-18T04:03:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "d93426b5-d0cd-461c-b03d-6cd238c57609", 2022-07-18T04:03:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 1682003393 2022-07-18T04:03:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:03:42.837 INFO:teuthology.orchestra.run.smithi159.stdout: "key": "ea63e6cf-483a-453e-81a7-bf1983b4a211", 2022-07-18T04:03:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: "val": 2112101669 2022-07-18T04:03:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:03:42.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.839 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zonegroup": "test-zonegroup", 2022-07-18T04:03:42.839 INFO:teuthology.orchestra.run.smithi159.stdout: "master_zone": "9b07f48a-ef73-4ad5-94be-e184c9a423d4", 2022-07-18T04:03:42.839 INFO:teuthology.orchestra.run.smithi159.stdout: "period_config": { 2022-07-18T04:03:42.839 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_quota": { 2022-07-18T04:03:42.840 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false, 2022-07-18T04:03:42.840 INFO:teuthology.orchestra.run.smithi159.stdout: "check_on_raw": false, 2022-07-18T04:03:42.840 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size": -1, 2022-07-18T04:03:42.841 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size_kb": 0, 2022-07-18T04:03:42.841 INFO:teuthology.orchestra.run.smithi159.stdout: "max_objects": -1 2022-07-18T04:03:42.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.841 INFO:teuthology.orchestra.run.smithi159.stdout: "user_quota": { 2022-07-18T04:03:42.842 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false, 2022-07-18T04:03:42.842 INFO:teuthology.orchestra.run.smithi159.stdout: "check_on_raw": false, 2022-07-18T04:03:42.842 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size": -1, 2022-07-18T04:03:42.842 INFO:teuthology.orchestra.run.smithi159.stdout: "max_size_kb": 0, 2022-07-18T04:03:42.843 INFO:teuthology.orchestra.run.smithi159.stdout: "max_objects": -1 2022-07-18T04:03:42.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.843 INFO:teuthology.orchestra.run.smithi159.stdout: "user_ratelimit": { 2022-07-18T04:03:42.843 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T04:03:42.844 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T04:03:42.844 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T04:03:42.844 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T04:03:42.844 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T04:03:42.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.845 INFO:teuthology.orchestra.run.smithi159.stdout: "bucket_ratelimit": { 2022-07-18T04:03:42.845 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T04:03:42.846 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T04:03:42.846 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T04:03:42.846 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T04:03:42.847 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T04:03:42.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.847 INFO:teuthology.orchestra.run.smithi159.stdout: "anonymous_ratelimit": { 2022-07-18T04:03:42.848 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_ops": 0, 2022-07-18T04:03:42.848 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_ops": 0, 2022-07-18T04:03:42.848 INFO:teuthology.orchestra.run.smithi159.stdout: "max_read_bytes": 0, 2022-07-18T04:03:42.848 INFO:teuthology.orchestra.run.smithi159.stdout: "max_write_bytes": 0, 2022-07-18T04:03:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: "enabled": false 2022-07-18T04:03:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:03:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:03:42.849 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_id": "66d17ce1-c2e4-4210-a2b7-a5e3888bc178", 2022-07-18T04:03:42.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_name": "test-realm", 2022-07-18T04:03:42.850 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:03:42.850 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:03:42.851 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2022-07-18T04:03:44.588 INFO:tasks.rgw.c1.client.1.smithi133.stdout:*** Caught signal (Segmentation fault) ** 2022-07-18T04:03:44.588 INFO:tasks.rgw.c1.client.1.smithi133.stdout: in thread 7f94b9401700 thread_name:rados_async 2022-07-18T04:03:44.589 INFO:tasks.rgw.c1.client.1.smithi133.stdout: ceph version 17.0.0-13595-gff10064a (ff10064ab878e58afb1d2538b2086a2d4a41c017) quincy (dev) 2022-07-18T04:03:44.589 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f94cc3ecce0] 2022-07-18T04:03:44.590 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2: [0x561939379400] 2022-07-18T04:03:44.590 INFO:tasks.rgw.c1.client.1.smithi133.stdout:2022-07-18T04:03:44.585+0000 7f94b9401700 -1 *** Caught signal (Segmentation fault) ** 2022-07-18T04:03:44.591 INFO:tasks.rgw.c1.client.1.smithi133.stdout: in thread 7f94b9401700 thread_name:rados_async 2022-07-18T04:03:44.592 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2022-07-18T04:03:44.592 INFO:tasks.rgw.c1.client.1.smithi133.stdout: ceph version 17.0.0-13595-gff10064a (ff10064ab878e58afb1d2538b2086a2d4a41c017) quincy (dev) 2022-07-18T04:03:44.592 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f94cc3ecce0] 2022-07-18T04:03:44.593 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2: [0x561939379400] 2022-07-18T04:03:44.593 INFO:tasks.rgw.c1.client.1.smithi133.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-07-18T04:03:44.594 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2022-07-18T04:03:44.602 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 0> 2022-07-18T04:03:44.585+0000 7f94b9401700 -1 *** Caught signal (Segmentation fault) ** 2022-07-18T04:03:44.602 INFO:tasks.rgw.c1.client.1.smithi133.stdout: in thread 7f94b9401700 thread_name:rados_async 2022-07-18T04:03:44.602 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2022-07-18T04:03:44.603 INFO:tasks.rgw.c1.client.1.smithi133.stdout: ceph version 17.0.0-13595-gff10064a (ff10064ab878e58afb1d2538b2086a2d4a41c017) quincy (dev) 2022-07-18T04:03:44.603 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f94cc3ecce0] 2022-07-18T04:03:44.604 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2: [0x561939379400] 2022-07-18T04:03:44.604 INFO:tasks.rgw.c1.client.1.smithi133.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-07-18T04:03:44.604 INFO:tasks.rgw.c1.client.1.smithi133.stdout: 2022-07-18T04:03:44.776 INFO:tasks.rgw.c1.client.1.smithi133.stderr:daemon-helper: command crashed with signal 11 2022-07-18T04:03:49.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-07-18T04:03:55.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2022-07-18T04:04:01.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~11s 2022-07-18T04:04:06.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~17s 2022-07-18T04:04:12.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~22s 2022-07-18T04:04:17.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~28s 2022-07-18T04:04:23.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~34s 2022-07-18T04:04:29.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~39s 2022-07-18T04:04:34.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2022-07-18T04:04:40.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2022-07-18T04:04:42.852 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-45 2022-07-18T04:04:42.853 DEBUG:boto:path=/mlvank-45/ 2022-07-18T04:04:42.854 DEBUG:boto:auth_path=/mlvank-45/ 2022-07-18T04:04:42.855 DEBUG:boto:Method: PUT 2022-07-18T04:04:42.855 DEBUG:boto:Path: /mlvank-45/ 2022-07-18T04:04:42.856 DEBUG:boto:Data: 2022-07-18T04:04:42.856 DEBUG:boto:Headers: {} 2022-07-18T04:04:42.857 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T04:04:42.857 DEBUG:boto:Port: 8000 2022-07-18T04:04:42.858 DEBUG:boto:Params: {} 2022-07-18T04:04:42.859 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T04:04:42.860 DEBUG:boto:Token: None 2022-07-18T04:04:42.862 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 04:04:42 GMT /mlvank-45/ 2022-07-18T04:04:42.863 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:PjKVqf0QlouJvdKsgecoPcUdUoo= 2022-07-18T04:04:42.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 04:04:42 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:PjKVqf0QlouJvdKsgecoPcUdUoo=', 'Content-Length': '0'} 2022-07-18T04:04:42.897 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc57bb14561b6666-0062d4dbda-118f-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 18 Jul 2022 04:04:42 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T04:04:42.898 DEBUG:boto:path=/mlvank-45/testredirect 2022-07-18T04:04:42.899 DEBUG:boto:auth_path=/mlvank-45/testredirect 2022-07-18T04:04:42.900 DEBUG:boto:Method: PUT 2022-07-18T04:04:42.900 DEBUG:boto:Path: /mlvank-45/testredirect 2022-07-18T04:04:42.900 DEBUG:boto:Data: 2022-07-18T04:04:42.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-07-18T04:04:42.902 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T04:04:42.902 DEBUG:boto:Port: 8000 2022-07-18T04:04:42.903 DEBUG:boto:Params: {} 2022-07-18T04:04:42.903 DEBUG:boto:Token: None 2022-07-18T04:04:42.904 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 18 Jul 2022 04:04:42 GMT /mlvank-45/testredirect 2022-07-18T04:04:42.904 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:ml4vkFjV+tO/SfSLj/o3OVg7tj8= 2022-07-18T04:04:42.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 18 Jul 2022 04:04:42 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:ml4vkFjV+tO/SfSLj/o3OVg7tj8='} 2022-07-18T04:04:42.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfbf6933a02d7b07-0062d4dbda-118f-test-zone1'), ('Date', 'Mon, 18 Jul 2022 04:04:42 GMT'), ('Connection', 'Keep-Alive')] 2022-07-18T04:04:42.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:42.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:42.925 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:04:42.977 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:04:42.978 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:04:43.024 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=c2173771-3b17-40d3-8c8b-0df64165bc30 2022-07-18T04:04:43.036 INFO:teuthology.orchestra.run.smithi133.stdout:[ 2022-07-18T04:04:43.037 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T04:04:43.037 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:04:43.038 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:04:43.073958Z" 2022-07-18T04:04:43.038 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T04:04:43.038 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T04:04:43.039 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "", 2022-07-18T04:04:43.040 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "0.000000" 2022-07-18T04:04:43.040 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T04:04:43.040 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T04:04:43.040 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:04:43.041 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T04:04:42.891928Z" 2022-07-18T04:04:43.041 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2022-07-18T04:04:43.041 INFO:teuthology.orchestra.run.smithi133.stdout: { 2022-07-18T04:04:43.041 INFO:teuthology.orchestra.run.smithi133.stdout: "marker": "1_1658113606.719799_131.1", 2022-07-18T04:04:43.042 INFO:teuthology.orchestra.run.smithi133.stdout: "last_update": "2022-07-18T03:06:46.719799Z" 2022-07-18T04:04:43.043 INFO:teuthology.orchestra.run.smithi133.stdout: } 2022-07-18T04:04:43.043 INFO:teuthology.orchestra.run.smithi133.stdout:] 2022-07-18T04:04:43.045 DEBUG:rgw_multi.tests:master meta markers={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:04:43.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-07-18T04:04:43.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:43.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:43.048 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:04:43.102 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:04:43.102 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:04:43.165 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:04:43.165 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:04:43.166 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:04:43.166 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:04:43.167 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:04:43.167 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:04:43.168 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:43.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:43.168 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:04:43.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:43.169 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:04:43.170 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:43.170 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:43.171 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:04:43.171 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:43.172 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:04:43.173 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:43.173 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:04:43.174 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:43.174 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:43.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:43.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:43.176 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:04:43.176 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:43.177 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:43.177 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:04:43.178 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:43.179 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:43.179 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:43.179 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:04:43.180 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:43.180 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:43.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:43.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:43.182 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:04:43.182 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:43.183 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:43.183 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:04:43.183 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:43.184 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:43.184 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:43.185 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:04:43.185 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:43.185 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:43.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:43.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:43.186 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:04:43.187 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:43.187 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:43.187 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:04:43.187 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:43.188 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:43.188 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:43.188 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:04:43.188 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:43.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:43.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:43.189 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:04:43.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:43.190 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:04:43.190 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:04:43.190 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:04:43.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:43.191 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:04:43Z" 2022-07-18T04:04:43.191 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:04:43.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:04:43Z" } 2022-07-18T04:04:43.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:04:43.194 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:04:43.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:04:43.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:04:45.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~56s 2022-07-18T04:04:48.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:48.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:48.199 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:04:48.253 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:04:48.254 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:04:48.322 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:04:48.323 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:04:48.323 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:04:48.324 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:04:48.324 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:04:48.325 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:04:48.325 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:48.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:48.326 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:04:48.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:48.327 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:04:48.328 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:48.328 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:48.328 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:04:48.329 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:48.329 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:04:48.330 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:48.330 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:04:48.330 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:48.331 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:48.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:48.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:48.332 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:04:48.332 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:48.332 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:48.333 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:04:48.333 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:48.333 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:48.333 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:48.334 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:04:48.334 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:48.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:48.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:48.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:48.335 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:04:48.335 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:48.335 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:48.336 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:04:48.336 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:48.336 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:48.336 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:48.337 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:04:48.337 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:48.337 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:48.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:48.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:48.338 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:04:48.338 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:48.338 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:48.339 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:04:48.339 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:48.339 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:48.339 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:48.340 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:04:48.340 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:48.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:48.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:48.341 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:04:48.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:48.341 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:04:48.341 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:04:48.342 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:04:48.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:48.342 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:04:48Z" 2022-07-18T04:04:48.342 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:04:48.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:04:48Z" } 2022-07-18T04:04:48.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:04:48.347 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:04:48.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:04:48.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:04:51.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~62s 2022-07-18T04:04:53.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:53.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:53.352 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:04:53.408 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:04:53.408 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:04:53.484 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:04:53.485 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:04:53.485 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:04:53.485 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:04:53.485 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:04:53.486 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:04:53.486 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:53.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:53.486 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:04:53.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:53.487 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:04:53.487 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:53.487 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:53.488 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:04:53.488 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:53.488 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:04:53.488 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:53.489 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:04:53.489 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:53.489 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:53.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:53.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:53.490 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:04:53.490 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:53.490 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:53.491 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:04:53.491 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:53.491 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:53.491 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:53.492 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:04:53.492 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:53.492 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:53.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:53.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:53.493 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:04:53.493 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:53.493 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:53.493 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:04:53.494 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:53.494 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:53.494 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:53.494 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:04:53.495 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:53.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:53.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:53.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:53.496 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:04:53.496 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:53.496 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:53.496 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:04:53.497 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:53.497 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:53.497 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:53.497 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:04:53.498 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:53.498 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:53.498 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:53.498 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:04:53.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:53.499 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:04:53.499 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:04:53.499 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:04:53.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:53.500 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:04:53Z" 2022-07-18T04:04:53.500 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:04:53.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:04:53Z" } 2022-07-18T04:04:53.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:04:53.503 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:04:53.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:04:53.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:04:57.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~67s 2022-07-18T04:04:58.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:58.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:04:58.507 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:04:58.563 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:04:58.563 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:04:58.635 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:04:58.636 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:04:58.637 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:04:58.637 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:04:58.638 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:04:58.638 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:04:58.639 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:58.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:58.640 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:04:58.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:58.641 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:04:58.641 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:58.642 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:58.642 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:04:58.643 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:58.643 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:04:58.644 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:58.644 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:04:58.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:58.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:58.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:58.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:58.647 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:04:58.647 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:58.648 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:58.648 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:04:58.649 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:58.649 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:58.650 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:58.650 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:04:58.651 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:58.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:58.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:58.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:58.653 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:04:58.653 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:58.654 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:58.654 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:04:58.655 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:58.655 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:58.656 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:58.656 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:04:58.657 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:58.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:58.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:58.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:04:58.659 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:04:58.659 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:04:58.660 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:04:58.660 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:04:58.661 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:04:58.661 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:04:58.662 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:04:58.662 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:04:58.663 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:04:58.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:58.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:04:58.664 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:04:58.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:58.665 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:04:58.665 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:04:58.666 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:04:58.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:04:58.667 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:04:58Z" 2022-07-18T04:04:58.667 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:04:58.669 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:04:58Z" } 2022-07-18T04:04:58.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:04:58.671 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:04:58.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:04:58.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:02.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~73s 2022-07-18T04:05:03.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:03.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:03.677 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:03.732 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:03.732 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:03.822 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:03.823 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:03.823 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:03.824 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:03.824 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:03.825 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:03.825 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:03.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:03.826 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:03.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:03.827 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:03.827 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:03.828 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:03.828 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:03.828 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:03.829 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:03.829 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:03.830 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:03.830 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:03.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:03.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:03.832 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:03.832 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:03.832 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:03.832 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:03.833 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:03.833 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:03.833 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:03.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:03.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:03.835 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:03.835 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:03.835 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:03.835 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:03.836 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:03.836 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:03.836 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:03.836 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:03.837 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:03.841 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:03.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:03.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:03.842 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:03.842 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:03.842 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:03.843 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:03.843 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:03.843 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:03.843 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:03.844 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:03.844 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:03.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:03.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:03.845 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:03.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:03.845 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:03.846 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:03.846 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:03.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:03.846 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:03Z" 2022-07-18T04:05:03.847 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:03.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:03Z" } 2022-07-18T04:05:03.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:03.849 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:03.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:03.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:08.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~78s 2022-07-18T04:05:08.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:08.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:08.853 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:08.909 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:08.909 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:08.972 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:08.973 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:08.973 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:08.973 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:08.974 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:08.974 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:08.974 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:08.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:08.975 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:08.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:08.975 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:08.975 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:08.976 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:08.976 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:08.976 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:08.976 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:08.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:08.977 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:08.977 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:08.977 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:08.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:08.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:08.978 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:08.979 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:08.979 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:08.979 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:08.980 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:08.980 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:08.980 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:08.980 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:08.981 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:08.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:08.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:08.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:08.982 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:08.982 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:08.982 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:08.982 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:08.983 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:08.983 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:08.983 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:08.983 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:08.984 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:08.984 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:08.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:08.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:08.985 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:08.985 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:08.985 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:08.985 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:08.986 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:08.986 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:08.986 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:08.986 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:08.987 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:08.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:08.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:08.987 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:08.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:08.988 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:08.988 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:08.988 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:08.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:08.989 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:08Z" 2022-07-18T04:05:08.989 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:08.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:08Z" } 2022-07-18T04:05:08.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:08.991 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:08.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:08.992 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:13.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~84s 2022-07-18T04:05:13.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:13.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:13.995 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:14.051 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:14.052 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:14.119 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:14.120 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:14.120 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:14.120 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:14.121 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:14.121 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:14.121 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:14.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:14.122 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:14.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:14.122 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:14.123 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:14.123 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:14.123 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:14.123 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:14.124 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:14.124 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:14.124 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:14.124 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:14.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:14.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:14.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:14.125 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:14.126 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:14.126 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:14.126 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:14.126 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:14.127 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:14.127 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:14.127 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:14.128 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:14.128 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:14.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:14.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:14.129 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:14.129 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:14.130 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:14.130 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:14.131 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:14.131 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:14.131 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:14.131 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:14.132 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:14.132 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:14.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:14.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:14.133 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:14.133 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:14.133 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:14.134 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:14.134 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:14.134 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:14.135 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:14.135 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:14.135 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:14.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:14.136 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:14.136 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:14.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:14.137 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:14.137 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:14.137 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:14.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:14.138 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:14Z" 2022-07-18T04:05:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:14.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:14Z" } 2022-07-18T04:05:14.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:14.141 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:14.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:14.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:19.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:19.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:19.146 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:19.200 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:19.200 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:19.267 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:19.268 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:19.268 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:19.268 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:19.269 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:19.269 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:19.269 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:19.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:19.270 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:19.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:19.271 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:19.271 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:19.271 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:19.271 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:19.272 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:19.272 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:19.272 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:19.272 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:19.273 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:19.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:19.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:19.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:19.274 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:19.274 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:19.274 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:19.275 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:19.275 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:19.275 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:19.275 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:19.276 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:19.276 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:19.276 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:19.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:19.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:19.277 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:19.277 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:19.277 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:19.278 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:19.278 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:19.278 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:19.278 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:19.279 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:19.279 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:19.279 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:19.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:19.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:19.280 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:19.280 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:19.280 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:19.281 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:19.281 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:19.281 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:19.281 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:19.282 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:19.282 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:19.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:19.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:19.283 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:19.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:19.283 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:19.284 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:19.284 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:19.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:19.284 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:19Z" 2022-07-18T04:05:19.285 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:19.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:19Z" } 2022-07-18T04:05:19.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:19.287 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:19.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:19.288 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:19.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2022-07-18T04:05:24.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:24.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:24.291 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:24.345 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:24.345 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:24.425 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:24.425 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:24.425 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:24.426 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:24.426 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:24.426 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:24.427 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:24.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:24.427 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:24.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:24.428 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:24.428 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:24.428 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:24.428 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:24.429 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:24.429 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:24.430 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:24.430 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:24.431 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:24.431 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:24.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:24.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:24.433 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:24.433 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:24.434 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:24.434 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:24.435 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:24.436 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:24.436 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:24.436 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:24.437 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:24.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:24.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:24.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:24.439 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:24.439 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:24.440 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:24.440 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:24.441 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:24.441 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:24.442 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:24.442 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:24.442 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:24.443 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:24.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:24.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:24.443 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:24.444 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:24.444 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:24.444 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:24.444 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:24.445 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:24.445 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:24.445 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:24.445 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:24.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:24.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:24.446 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:24.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:24.447 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:24.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:24.447 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:24.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:24.448 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:24Z" 2022-07-18T04:05:24.448 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:24.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:24Z" } 2022-07-18T04:05:24.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:24.451 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:24.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:24.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:25.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2022-07-18T04:05:29.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:29.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:29.456 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:29.512 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:29.512 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:29.584 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:29.584 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:29.584 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:29.585 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:29.585 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:29.585 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:29.585 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:29.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:29.586 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:29.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:29.587 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:29.587 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:29.587 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:29.587 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:29.588 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:29.588 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:29.588 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:29.588 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:29.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:29.589 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:29.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:29.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:29.590 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:29.590 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:29.590 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:29.590 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:29.591 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:29.591 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:29.591 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:29.591 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:29.592 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:29.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:29.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:29.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:29.593 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:29.593 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:29.593 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:29.593 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:29.594 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:29.594 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:29.594 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:29.594 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:29.595 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:29.595 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:29.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:29.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:29.596 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:29.596 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:29.596 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:29.597 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:29.597 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:29.597 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:29.597 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:29.598 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:29.598 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:29.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:29.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:29.599 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:29.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:29.599 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:29.600 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:29.600 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:29.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:29.600 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:29Z" 2022-07-18T04:05:29.601 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:29.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:29Z" } 2022-07-18T04:05:29.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:29.603 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:29.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:29.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:30.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~101s 2022-07-18T04:05:34.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:34.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:34.607 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:34.664 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:34.664 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:34.751 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:34.751 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:34.751 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:34.752 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:34.752 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:34.752 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:34.752 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:34.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:34.753 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:34.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:34.753 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:34.754 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:34.754 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:34.754 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:34.755 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:34.755 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:34.755 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:34.756 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:34.756 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:34.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:34.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:34.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:34.757 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:34.757 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:34.757 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:34.758 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:34.758 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:34.758 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:34.758 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:34.759 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:34.759 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:34.759 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:34.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:34.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:34.760 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:34.760 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:34.760 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:34.761 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:34.761 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:34.761 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:34.761 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:34.762 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:34.762 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:34.762 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:34.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:34.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:34.763 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:34.763 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:34.764 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:34.764 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:34.764 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:34.764 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:34.765 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:34.765 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:34.765 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:34.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:34.766 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:34.766 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:34.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:34.766 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:34.767 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:34.767 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:34.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:34.767 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:34Z" 2022-07-18T04:05:34.768 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:34.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:34Z" } 2022-07-18T04:05:34.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:34.770 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:34.771 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:34.771 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:36.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~107s 2022-07-18T04:05:39.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:39.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:39.774 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:39.830 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:39.831 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:39.931 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:39.932 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:39.933 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:39.934 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:39.935 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:39.936 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:39.938 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:39.939 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:39.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:39.941 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658116712.851438_288.1", 2022-07-18T04:05:39.942 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:39.942 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:39.942 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:39.942 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:58:32.851438Z", 2022-07-18T04:05:39.943 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:39.943 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:39.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:39.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:39.944 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:39.944 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:39.944 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:39.944 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:39.945 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:39.945 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:39.945 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:39.945 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:39.946 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:39.946 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:39.946 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:39.947 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:39.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:39.947 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:39.947 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:39.948 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:39.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:39.948 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:39Z" 2022-07-18T04:05:39.949 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:39.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658116712.851438_288.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:58:32.851438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:39Z" } 2022-07-18T04:05:39.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658116712.851438_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:58:32.851438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:39.951 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:39.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658116712.851438_288.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:39.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1658117082.891928_311.1 target=1_1658116712.851438_288.1, shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:41.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~112s 2022-07-18T04:05:44.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:44.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:44.955 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:45.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:45.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:45.072 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:45.073 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:45.073 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:45.073 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:45.074 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:45.074 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:45.074 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:45.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:45.075 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:45.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:45.075 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:45.075 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:45.076 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:45.076 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:45.076 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:45.076 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:45.077 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:45.077 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:45.077 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:45.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:45.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:45.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:45.078 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:45.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:45.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:45.079 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:45.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:45.079 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:45.080 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:45.080 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:45.080 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:45.080 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:45.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:45.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:45.081 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:45.081 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:45.081 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:45.082 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:05:45.082 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:45.083 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:45.083 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:45.083 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:05:45.083 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:45.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:45.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:45.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:45.084 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:45.085 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:45.085 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:45.085 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:45.086 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:45.086 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:45.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:45.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:45.087 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:45.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:45.088 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:45.088 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:45.088 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:45.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:45.089 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:45Z" 2022-07-18T04:05:45.089 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:45.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:45Z" } 2022-07-18T04:05:45.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:45.092 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:45.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:45.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:47.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~118s 2022-07-18T04:05:50.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:50.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:50.095 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:50.147 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:50.147 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:50.238 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:50.239 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:50.239 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:50.239 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:50.240 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:50.240 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:50.240 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:50.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:50.241 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:50.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:50.241 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:50.242 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:50.242 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:50.242 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:50.243 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:50.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:50.245 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:50.246 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:50.246 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:50.246 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:50.247 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:50.247 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:50.247 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:50.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:50.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:50.248 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:50.248 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:05:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:50.249 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:05:50.250 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:50.251 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:50.252 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:50.253 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:50.255 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:50.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:50.256 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:50Z" 2022-07-18T04:05:50.256 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:50.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:50Z" } 2022-07-18T04:05:50.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:50.259 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:50.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:50.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:53.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~123s 2022-07-18T04:05:55.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:55.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:05:55.262 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:05:55.317 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:05:55.318 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:05:55.402 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:05:55.403 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:05:55.403 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:05:55.403 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:05:55.404 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:05:55.404 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:05:55.404 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:55.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:55.405 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:05:55.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:55.405 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:05:55.406 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:55.406 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:55.406 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:05:55.406 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:55.407 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:05:55.407 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:55.407 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:05:55.407 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:55.408 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:55.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:55.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:55.409 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:05:55.409 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:55.409 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:55.409 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:05:55.410 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:55.410 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:55.410 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:55.410 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:05:55.411 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:55.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:55.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:55.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:55.412 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:05:55.412 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:55.412 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:55.412 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:05:55.413 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:55.413 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:55.413 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:55.413 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:05:55.414 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:55.414 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:55.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:55.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:05:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:05:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:05:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:05:55.415 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:05:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:05:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:05:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:05:55.416 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:05:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:05:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:05:55.417 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:05:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:05:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:05:55.418 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:05:55.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:05:55.419 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:05:55Z" 2022-07-18T04:05:55.419 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:05:55.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:05:55Z" } 2022-07-18T04:05:55.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:05:55.421 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:05:55.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:05:55.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:05:58.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~129s 2022-07-18T04:06:00.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:00.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:00.425 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:00.481 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:00.482 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:00.548 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:00.549 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:00.549 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:00.549 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:00.549 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:00.550 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:00.550 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:00.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:00.551 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:00.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:00.551 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:00.551 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:00.551 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:00.552 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:00.552 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:00.552 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:00.552 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:00.553 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:00.553 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:00.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:00.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:00.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:00.554 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:00.554 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:00.554 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:00.555 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:00.555 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:00.555 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:00.556 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:00.556 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:00.556 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:00.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:00.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:00.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:00.557 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:00.557 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:00.557 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:00.558 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:00.558 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:00.558 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:00.558 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:00.559 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:00.559 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:00.559 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:00.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:00.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:00.560 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:00.560 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:00.560 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:00.562 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:00.562 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:00.562 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:00.563 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:00.563 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:00.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:00.563 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:00.564 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:00.564 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:00.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:00.564 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:00Z" 2022-07-18T04:06:00.565 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:00.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:00Z" } 2022-07-18T04:06:00.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:00.567 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:00.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:00.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:04.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2022-07-18T04:06:05.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:05.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:05.571 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:05.626 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:05.627 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:05.696 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:05.697 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:05.697 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:05.697 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:05.698 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:05.698 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:05.698 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:05.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:05.699 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:05.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:05.699 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:05.700 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:05.700 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:05.700 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:05.701 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:05.701 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:05.701 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:05.701 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:05.702 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:05.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:05.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:05.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:05.703 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:05.703 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:05.703 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:05.704 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:05.704 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:05.704 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:05.704 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:05.705 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:05.705 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:05.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:05.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:05.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:05.706 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:05.706 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:05.707 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:05.707 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:05.707 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:05.708 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:05.708 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:05.708 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:05.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:05.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:05.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:05.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:05.710 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:05.710 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:05.710 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:05.710 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:05.712 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:05.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:05.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:05.712 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:05.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:05.713 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:05.713 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:05.713 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:05.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:05.714 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:05Z" 2022-07-18T04:06:05.714 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:05.716 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:05Z" } 2022-07-18T04:06:05.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:05.717 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:05.717 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:05.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:09.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2022-07-18T04:06:10.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:10.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:10.720 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:10.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:10.775 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:10.838 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:10.838 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:10.839 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:10.839 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:10.839 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:10.840 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:10.840 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:10.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:10.840 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:10.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:10.841 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:10.841 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:10.841 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:10.842 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:10.842 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:10.842 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:10.842 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:10.843 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:10.843 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:10.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:10.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:10.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:10.844 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:10.844 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:10.844 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:10.845 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:10.845 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:10.845 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:10.845 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:10.846 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:10.846 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:10.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:10.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:10.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:10.847 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:10.847 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:10.847 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:10.848 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:10.848 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:10.848 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:10.849 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:10.849 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:10.849 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:10.849 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:10.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:10.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:10.850 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:10.850 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:10.851 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:10.851 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:10.851 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:10.851 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:10.852 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:10.852 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:10.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:10.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:10.853 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:10.853 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:10.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:10.853 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:10.854 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:10.854 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:10.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:10.855 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:10Z" 2022-07-18T04:06:10.855 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:10.856 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:10Z" } 2022-07-18T04:06:10.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:10.857 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:10.858 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:10.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:15.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~146s 2022-07-18T04:06:15.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:15.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:15.861 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:15.917 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:15.917 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:15.989 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:15.989 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:15.990 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:15.990 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:15.990 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:15.990 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:15.991 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:15.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:15.991 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:15.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:15.992 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:15.992 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:15.992 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:15.993 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:15.993 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:15.993 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:15.994 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:15.994 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:15.994 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:15.995 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:15.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:15.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:15.996 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:15.996 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:15.996 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:15.997 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:15.997 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:15.998 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:15.999 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:15.999 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:15.999 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:16.000 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:16.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:16.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:16.001 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:16.002 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:16.002 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:16.002 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:16.003 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:16.003 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:16.004 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:16.004 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:16.004 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:16.005 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:16.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:16.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:16.005 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:16.006 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:16.006 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:16.006 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:16.007 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:16.007 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:16.007 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:16.007 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:16.008 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:16.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:16.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:16.008 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:16.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:16.009 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:16.009 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:16.009 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:16.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:16.010 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:15Z" 2022-07-18T04:06:16.010 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:16.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:15Z" } 2022-07-18T04:06:16.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:16.013 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:16.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:16.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:21.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:21.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:21.017 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:21.073 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:21.074 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:21.149 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:21.149 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:21.150 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:21.150 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:21.150 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:21.151 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:21.151 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:21.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:21.151 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:21.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:21.152 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:21.152 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:21.153 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:21.153 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:21.153 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:21.153 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:21.154 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:21.154 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:21.154 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:21.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:21.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:21.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:21.156 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:21.156 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:21.156 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:21.156 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:21.157 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:21.157 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:21.158 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:21.158 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:21.158 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:21.158 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:21.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:21.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:21.159 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:21.160 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:21.160 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:21.160 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:21.161 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:21.161 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:21.161 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:21.161 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:21.162 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:21.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:21.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:21.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:21.163 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:21.163 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:21.163 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:21.164 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:21.164 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:21.164 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:21.164 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:21.165 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:21.165 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:21.165 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:21.165 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:21.166 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:21.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:21.166 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:21.167 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:21.167 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:21.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:21.167 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:21Z" 2022-07-18T04:06:21.168 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:21.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~151s 2022-07-18T04:06:21.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:21Z" } 2022-07-18T04:06:21.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:21.171 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:21.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:21.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:26.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:26.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:26.175 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:26.233 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:26.233 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:26.304 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:26.304 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:26.304 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:26.305 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:26.305 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:26.305 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:26.306 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:26.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:26.306 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:26.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:26.307 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:26.307 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:26.307 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:26.307 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:26.308 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:26.308 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:26.308 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:26.308 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:26.309 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:26.309 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:26.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:26.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:26.309 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:26.310 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:26.310 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:26.310 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:26.311 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:26.311 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:26.311 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:26.311 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:26.312 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:26.312 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:26.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:26.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:26.313 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:26.313 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:26.313 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:26.313 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:26.313 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:26.314 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:26.314 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:26.314 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:26.314 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:26.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:26.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:26.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:26.315 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:26.316 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:26.316 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:26.316 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:26.316 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:26.317 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:26.317 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:26.317 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:26.317 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:26.318 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:26.318 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:26.318 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:26.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:26.319 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:26.319 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:26.319 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:26.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:26.320 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:26Z" 2022-07-18T04:06:26.320 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:26.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:26Z" } 2022-07-18T04:06:26.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:26.322 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:26.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:26.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:26.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~157s 2022-07-18T04:06:31.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:31.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:31.326 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:31.381 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:31.381 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:31.443 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:31Z" 2022-07-18T04:06:31.460 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:31.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:31Z" } 2022-07-18T04:06:31.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:31.463 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:31.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:31.464 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:32.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~163s 2022-07-18T04:06:36.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:36.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:36.467 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:36.523 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:36.523 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:36.588 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:36.593 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:36.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:36.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:36.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:36.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:36.594 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:36.594 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:36.594 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:36.595 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:36.595 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:36.595 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:36.595 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:36.596 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:36.596 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:36.596 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:36.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:36.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:36.597 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:36.597 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:36.598 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:36.598 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:36.598 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:36.598 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:36.599 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:36.599 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:36.599 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:36.599 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:36.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:36.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:36.600 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:36.601 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:36.601 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:36.601 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:36.601 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:36.601 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:36.602 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:36.602 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:36.602 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:36.603 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:36.603 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:36.603 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:36.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:36.604 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:36.604 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:36.604 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:36.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:36.605 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:36Z" 2022-07-18T04:06:36.605 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:36.606 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:36Z" } 2022-07-18T04:06:36.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:36.607 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:36.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:36.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:37.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~168s 2022-07-18T04:06:41.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:41.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:41.611 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:41.666 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:41.667 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:41.743 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:41.743 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:41.743 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:41.744 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:41.744 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:41.744 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:41.744 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:41.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:41.745 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:41.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:41.746 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:41.746 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:41.746 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:41.746 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:41.747 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:41.747 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:41.747 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:41.747 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:41.748 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:41.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:41.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:41.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:41.749 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:41.749 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:41.749 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:41.749 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:41.750 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:41.750 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:41.750 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:41.751 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:41.751 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:41.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:41.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:41.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:41.752 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:41.752 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:41.752 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:41.753 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:41.753 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:41.753 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:41.753 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:41.754 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:41.754 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:41.754 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:41.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:41.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:41.755 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:41.755 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:41.755 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:41.756 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:41.756 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:41.756 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:41.756 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:41.757 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:41.757 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:41.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:41.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:41.758 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:41.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:41.758 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:41.759 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:41.759 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:41.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:41.759 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:41Z" 2022-07-18T04:06:41.760 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:41.761 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:41Z" } 2022-07-18T04:06:41.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:41.762 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:41.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:41.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:43.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~174s 2022-07-18T04:06:46.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:46.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:46.766 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:46.822 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:46.823 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:46.895 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:46.895 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:46.895 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:46.896 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:46.896 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:46.896 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:46.896 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:46.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:46.897 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:46.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:46.898 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:46.898 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:46.898 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:46.898 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:46.899 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:46.899 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:46.899 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:46.899 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:46.900 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:46.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:46.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:46.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:46.901 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:46.901 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:46.901 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:46.901 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:46.902 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:46.902 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:46.902 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:46.902 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:46.904 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:46.904 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:46.904 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:46.904 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:46.905 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:46.905 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:46.905 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:46.905 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:46.906 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:46.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:46.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:46.906 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:46.907 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:46.907 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:46.907 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:46.907 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:46.908 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:46.908 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:46.908 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:46.908 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:46.909 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:46.909 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:46.909 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:46.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:46.910 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:46.910 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:46.910 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:46.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:46.911 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:46Z" 2022-07-18T04:06:46.911 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:46.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:46Z" } 2022-07-18T04:06:46.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:46.913 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:46.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:46.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:49.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~179s 2022-07-18T04:06:51.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:51.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:51.917 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:51.971 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:51.972 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:52.033 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:52.033 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:52.034 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:52.034 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:52.034 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:52.035 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:52.035 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:52.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:52.035 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:52.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:52.036 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:52.036 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:52.036 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:52.037 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:52.037 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:52.037 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:52.038 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:52.038 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:52.038 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:52.038 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:52.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:52.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:52.039 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:52.039 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:52.040 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:52.040 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:52.040 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:52.040 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:52.041 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:52.041 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:52.041 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:52.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:52.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:52.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:52.042 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:52.043 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:52.043 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:52.043 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:52.043 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:52.044 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:52.044 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:52.044 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:52.044 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:52.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:52.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:52.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:52.045 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:52.046 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:52.046 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:52.047 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:52.047 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:52.047 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:52.047 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:52.048 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:52.048 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:52.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:52.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:52.049 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:52.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:52.049 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:52.049 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:52.050 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:52.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:52.050 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:52Z" 2022-07-18T04:06:52.050 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:52.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:52Z" } 2022-07-18T04:06:52.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:52.053 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:52.054 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:52.054 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:06:54.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2022-07-18T04:06:57.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:57.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:06:57.057 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:06:57.113 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:06:57.114 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:06:57.198 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:06:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:06:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:06:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:06:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:06:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:06:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:06:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:06:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:06:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:06:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:06:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:06:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:06:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:06:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:06:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:06:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:06:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:06:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:06:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:06:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:06:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:06:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:06:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:06:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:06:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:06:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:06:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:06:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:06:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:06:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:06:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:06:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:06:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:06:57Z" 2022-07-18T04:06:57.214 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:06:57.215 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:06:57Z" } 2022-07-18T04:06:57.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:06:57.217 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:06:57.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:06:57.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:00.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~191s 2022-07-18T04:07:02.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:02.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:02.220 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:02.276 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:02.277 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:02.336 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:02.337 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:02.337 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:02.338 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:02.338 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:02.338 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:02.339 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:02.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:02.339 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:02.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:02.340 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:02.340 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:02.341 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:02.341 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:02.341 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:02.342 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:02.342 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:02.342 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:02.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:02.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:02.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:02.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:02.344 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:02.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:02.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:02.345 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:02.345 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:02.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:02.346 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:02.346 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:02.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:02.346 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:02.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:02.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:02.347 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:02.347 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:02.348 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:02.348 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:02.348 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:02.349 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:02.349 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:02.349 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:02.349 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:02.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:02.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:02.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:02.350 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:02.351 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:02.351 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:02.351 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:02.351 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:02.352 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:02.352 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:02.352 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:02.353 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:02.353 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:02.353 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:02.353 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:02.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:02.354 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:02.354 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:02.355 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:02.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:02.355 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:02Z" 2022-07-18T04:07:02.356 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:02.357 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:02Z" } 2022-07-18T04:07:02.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:02.358 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:02.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:02.359 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:06.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~196s 2022-07-18T04:07:07.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:07.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:07.362 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:07.418 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:07.418 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:07.488 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:07.488 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:07.489 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:07.489 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:07.489 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:07.489 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:07.490 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:07.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:07.490 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:07.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:07.491 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:07.491 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:07.491 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:07.492 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:07.492 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:07.492 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:07.492 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:07.493 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:07.493 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:07.493 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:07.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:07.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:07.494 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:07.494 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:07.494 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:07.495 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:07.495 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:07.495 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:07.495 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:07.496 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:07.496 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:07.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:07.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:07.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:07.497 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:07.497 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:07.497 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:07.498 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:07.498 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:07.498 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:07.498 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:07.499 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:07.499 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:07.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:07.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:07.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:07.500 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:07.500 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:07.500 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:07.501 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:07.501 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:07.501 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:07.501 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:07.502 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:07.502 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:07.503 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:07.503 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:07.503 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:07.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:07.504 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:07.504 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:07.504 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:07.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:07.505 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:07Z" 2022-07-18T04:07:07.505 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:07.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:07Z" } 2022-07-18T04:07:07.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:07.508 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:07.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:07.509 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:11.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~202s 2022-07-18T04:07:12.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:12.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:12.512 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:12.567 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:12.567 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:12.640 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:12.640 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:12.641 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:12.641 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:12.641 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:12.642 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:12.642 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:12.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:12.642 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:12.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:12.643 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:12.643 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:12.643 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:12.644 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:12.644 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:12.644 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:12.645 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:12.645 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:12.645 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:12.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:12.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:12.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:12.646 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:12.646 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:12.647 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:12.647 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:12.647 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:12.647 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:12.648 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:12.648 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:12.648 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:12.648 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:12.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:12.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:12.649 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:12.649 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:12.650 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:12.650 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:12.650 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:12.650 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:12.651 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:12.651 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:12.651 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:12.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:12.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:12.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:12.652 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:12.653 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:12.653 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:12.653 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:12.653 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:12.654 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:12.654 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:12.654 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:12.654 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:12.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:12.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:12.655 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:12.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:12.656 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:12.656 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:12.656 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:12.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:12.657 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:12Z" 2022-07-18T04:07:12.657 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:12.658 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:12Z" } 2022-07-18T04:07:12.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:12.659 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:12.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:12.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:17.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~207s 2022-07-18T04:07:17.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:17.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:17.663 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:17.731 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:17.732 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:17.865 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:17Z" 2022-07-18T04:07:17.884 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:17.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:17Z" } 2022-07-18T04:07:17.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:17.887 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:17.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:17.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:22.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~213s 2022-07-18T04:07:22.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:22.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:22.892 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:22.948 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:22.949 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:23.040 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:23.049 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:23.049 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:23Z" 2022-07-18T04:07:23.068 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:23.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:23Z" } 2022-07-18T04:07:23.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:23.069 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:23.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:23.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:28.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:28.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:28.073 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:28.128 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:28.129 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:28.207 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:28Z" 2022-07-18T04:07:28.228 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:28.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:28Z" } 2022-07-18T04:07:28.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:28.231 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:28.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:28.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:28.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~219s 2022-07-18T04:07:33.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:33.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:33.235 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:33.290 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:33.291 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:33.370 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:33.370 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:33.370 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:33.371 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:33.371 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:33.371 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:33.371 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:33.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:33.372 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:33.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:33.373 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:33.373 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:33.374 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:33.374 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:33.374 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:33.374 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:33.375 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:33.375 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:33.375 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:33.375 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:33.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:33.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:33.376 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:33.376 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:33.377 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:33.377 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:33.377 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:33.378 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:33.378 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:33.378 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:33.379 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:33.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:33.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:33.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:33.380 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:33.380 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:33.381 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:33.381 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:33.381 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:33.381 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:33.382 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:33.382 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:33.382 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:33.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:33.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:33.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:33.383 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:33.383 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:33.384 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:33.384 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:33.384 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:33.385 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:33.385 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:33.385 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:33.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:33.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:33.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:33.386 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:33.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:33.387 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:33.387 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:33.387 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:33.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:33.388 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:33Z" 2022-07-18T04:07:33.388 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:33.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:33Z" } 2022-07-18T04:07:33.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:33.391 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:33.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:33.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:34.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~224s 2022-07-18T04:07:38.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:38.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:38.394 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:38.445 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:38.446 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:38.510 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:38.511 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:38.512 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:38.512 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:38.513 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:38.513 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:38.514 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:38.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:38.515 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:38.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:38.515 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:38.516 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:38.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:38.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:38.517 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:38.517 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:38.518 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:38.518 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:38.519 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:38.519 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:38.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:38.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:38.520 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:38.520 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:38.520 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:38.521 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:38.521 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:38.521 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:38.521 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:38.522 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:38.522 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:38.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:38.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:38.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:38.523 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:38.523 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:38.524 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:38.524 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:38.524 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:38.524 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:38.525 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:38.525 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:38.525 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:38.525 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:38.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:38.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:38.526 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:38.526 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:38.527 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:38.527 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:38.527 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:38.527 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:38.528 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:38.528 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:38.528 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:38.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:38.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:38.529 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:38.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:38.529 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:38.530 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:38.530 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:38.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:38.530 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:38Z" 2022-07-18T04:07:38.531 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:38.532 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:38Z" } 2022-07-18T04:07:38.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:38.533 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:38.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:38.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:39.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2022-07-18T04:07:43.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:43.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:43.537 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:43.589 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:43.590 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:43.652 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:43.653 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:43.653 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:43.653 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:43.654 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:43.654 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:43.654 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:43.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:43.655 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:43.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:43.655 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:43.656 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:43.656 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:43.657 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:43.657 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:43.657 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:43.657 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:43.658 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:43.658 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:43.658 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:43.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:43.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:43.659 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:43.659 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:43.660 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:43.660 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:43.660 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:43.660 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:43.661 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:43.661 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:43.661 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:43.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:43.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:43.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:43.662 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:43.662 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:43.663 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:43.663 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:43.663 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:43.663 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:43.664 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:43.664 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:43.664 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:43.664 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:43.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:43.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:43.665 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:43.666 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:43.666 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:43.666 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:43.667 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:43.667 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:43.667 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:43.667 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:43.668 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:43.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:43.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:43.668 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:43.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:43.669 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:43.669 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:43.670 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:43.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:43.670 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:43Z" 2022-07-18T04:07:43.670 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:43.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:43Z" } 2022-07-18T04:07:43.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:43.673 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:43.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:43.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:45.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2022-07-18T04:07:48.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:48.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:48.677 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:48.732 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:48.733 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:48.803 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:48.804 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:48.804 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:48.805 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:48.805 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:48.805 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:48.806 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:48.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:48.806 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:48.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:48.807 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:48.807 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:48.808 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:48.808 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:48.808 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:48.808 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:48.809 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:48.809 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:48.809 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:48.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:48.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:48.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:48.810 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:48.810 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:48.811 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:48.811 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:48.811 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:48.812 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:48.812 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:48.812 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:48.812 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:48.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:48.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:48.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:48.813 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:48.814 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:48.814 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:48.814 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:48.814 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:48.815 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:48.815 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:48.815 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:48.816 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:48.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:48.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:48.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:48.817 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:48.817 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:48.817 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:48.817 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:48.818 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:48.818 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:48.818 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:48.818 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:48.819 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:48.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:48.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:48.820 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:48.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:48.820 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:48.821 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:48.821 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:48.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:48.822 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:48Z" 2022-07-18T04:07:48.823 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:48.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:48Z" } 2022-07-18T04:07:48.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:48.826 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:48.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:48.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:50.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~241s 2022-07-18T04:07:53.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:53.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:53.829 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:53.885 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:53.885 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:53.972 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:53.972 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:53.973 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:53.973 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:53.973 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:53.973 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:53.974 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:53.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:53.974 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:53.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:53.975 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:53.975 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:53.975 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:53.975 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:53.976 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:53.976 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:53.976 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:53.976 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:53.977 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:53.977 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:53.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:53.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:53.978 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:53.978 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:53.978 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:53.978 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:53.979 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:53.979 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:53.979 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:53.979 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:53.979 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:53.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:53.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:53.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:53.980 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:53.981 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:53.981 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:53.981 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:53.981 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:53.982 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:53.982 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:53.982 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:53.982 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:53.983 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:53.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:53.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:53.983 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:53.984 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:53.984 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:53.984 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:53.984 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:53.985 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:53.985 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:53.985 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:53.985 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:53.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:53.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:53.986 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:53.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:53.987 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:53.987 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:53.987 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:53.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:53.987 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:53Z" 2022-07-18T04:07:53.988 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:53.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:53Z" } 2022-07-18T04:07:53.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:53.990 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:53.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:53.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:07:56.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~247s 2022-07-18T04:07:58.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:58.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:07:58.994 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:07:59.052 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:07:59.052 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:07:59.134 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:07:59.135 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:07:59.135 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:07:59.135 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:07:59.136 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:07:59.136 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:07:59.136 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:59.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:59.137 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:07:59.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:59.137 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:07:59.137 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:59.138 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:59.138 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:07:59.138 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:59.138 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:07:59.139 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:59.139 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:07:59.139 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:59.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:59.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:59.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:59.140 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:07:59.140 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:59.141 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:59.141 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:07:59.141 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:59.141 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:59.142 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:59.142 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:07:59.142 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:59.143 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:59.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:59.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:59.143 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:07:59.144 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:59.144 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:59.144 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:07:59.144 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:59.145 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:59.145 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:59.145 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:07:59.145 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:59.145 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:59.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:59.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:07:59.146 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:07:59.146 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:07:59.147 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:07:59.147 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:07:59.147 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:07:59.147 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:07:59.148 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:07:59.148 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:07:59.148 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:07:59.148 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:59.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:07:59.149 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:07:59.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:59.150 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:07:59.150 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:07:59.150 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:07:59.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:07:59.151 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:07:59Z" 2022-07-18T04:07:59.151 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:07:59.152 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:07:59Z" } 2022-07-18T04:07:59.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:07:59.153 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:07:59.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:07:59.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:02.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~252s 2022-07-18T04:08:04.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:04.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:04.157 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:04.210 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:04.210 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:04.281 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:04.281 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:04.282 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:04.282 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:04.282 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:04.282 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:04.283 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:04.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:04.283 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:04.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:04.284 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:04.284 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:04.284 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:04.285 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:04.285 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:04.285 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:04.285 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:04.286 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:04.286 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:04.286 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:04.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:04.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:04.287 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:04.288 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:04.288 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:04.288 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:04.288 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:04.289 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:04.289 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:04.289 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:04.289 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:04.290 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:04.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:04.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:04.291 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:04.291 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:04.292 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:04.292 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:04.293 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:04.293 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:04.293 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:04.294 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:04.294 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:04.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:04.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:04.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:04.295 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:04.295 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:04.295 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:04.296 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:04.296 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:04.296 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:04.296 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:04.297 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:04.297 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:04.297 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:04.297 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:04.298 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:04.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:04.298 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:04.299 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:04.299 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:04.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:04.300 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:04Z" 2022-07-18T04:08:04.300 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:04.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:04Z" } 2022-07-18T04:08:04.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:04.303 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:04.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:04.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:07.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~258s 2022-07-18T04:08:09.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:09.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:09.307 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:09.362 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:09.363 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:09.428 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:09.428 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:09.429 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:09.429 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:09.430 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:09.430 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:09.431 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:09.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:09.432 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:09.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:09.433 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:09.433 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:09.434 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:09.434 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:09.434 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:09.434 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:09.435 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:09.435 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:09.436 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:09.436 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:09.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:09.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:09.437 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:09.437 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:09.437 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:09.438 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:09.438 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:09.438 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:09.439 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:09.439 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:09.439 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:09.439 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:09.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:09.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:09.440 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:09.440 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:09.441 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:09.441 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:09.441 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:09.442 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:09.442 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:09.442 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:09.442 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:09.443 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:09.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:09.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:09.443 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:09.444 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:09.444 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:09.444 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:09.444 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:09.445 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:09.445 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:09.445 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:09.445 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:09.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:09.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:09.446 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:09.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:09.447 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:09.447 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:09.447 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:09.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:09.448 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:09Z" 2022-07-18T04:08:09.448 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:09.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:09Z" } 2022-07-18T04:08:09.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:09.451 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:09.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:09.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:13.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~263s 2022-07-18T04:08:14.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:14.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:14.454 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:14.510 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:14.510 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:14.584 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:14.585 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:14.585 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:14.586 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:14.586 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:14.587 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:14.587 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:14.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:14.588 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:14.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:14.588 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:14.588 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:14.589 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:14.590 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:14.590 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:14.591 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:14.591 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:14.592 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:14.593 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:14.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:14.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:14.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:14.595 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:14.595 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:14.596 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:14.596 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:14.597 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:14.597 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:14.598 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:14.598 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:14.599 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:14.599 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:14.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:14.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:14.601 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:14.601 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:14.602 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:14.602 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:14.603 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:14.603 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:14.604 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:14.604 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:14.605 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:14.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:14.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:14.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:14.607 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:14.607 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:14.608 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:14.608 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:14.609 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:14.609 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:14.610 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:14.610 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:14.611 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:14.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:14.612 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:14.612 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:14.613 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:14.613 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:14.614 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:14.614 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:14.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:14.615 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:14Z" 2022-07-18T04:08:14.616 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:14.617 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:14Z" } 2022-07-18T04:08:14.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:14.618 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:14.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:14.620 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:18.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~269s 2022-07-18T04:08:19.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:19.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:19.622 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:19.678 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:19.678 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:19.753 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:19.754 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:19.754 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:19.755 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:19.755 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:19.756 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:19.756 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:19.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:19.758 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:19.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:19.759 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:19.759 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:19.760 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:19.760 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:19.761 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:19.762 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:19.762 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:19.763 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:19.763 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:19.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:19.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:19.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:19.765 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:19.766 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:19.767 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:19.767 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:19.768 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:19.768 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:19.769 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:19.769 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:19.770 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:19.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:19.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:19.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:19.772 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:19.773 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:19.773 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:19.774 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:19.774 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:19.775 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:19.776 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:19.776 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:19.777 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:19.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:19.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:19.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:19.779 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:19.780 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:19.780 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:19.781 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:19.782 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:19.782 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:19.783 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:19.783 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:19.784 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:19.784 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:19.785 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:19.785 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:19.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:19.786 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:19.786 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:19.787 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:19.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:19.787 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:19Z" 2022-07-18T04:08:19.787 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:19.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:19Z" } 2022-07-18T04:08:19.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:19.790 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:19.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:19.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:24.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2022-07-18T04:08:24.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:24.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:24.794 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:24.842 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:24.843 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:24.925 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:24.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:24.938 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:24.938 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:24.938 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:24.939 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:24.939 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:24.939 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:24.939 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:24.940 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:24.940 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:24.940 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:24.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:24.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:24.941 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:24.941 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:24.941 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:24.942 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:24.942 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:24.943 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:24.943 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:24.943 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:24.943 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:24.944 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:24.944 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:24.944 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:24.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:24.944 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:24.945 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:24.945 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:24.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:24.946 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:24Z" 2022-07-18T04:08:24.947 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:24.948 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:24Z" } 2022-07-18T04:08:24.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:24.950 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:24.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:24.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:29.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:29.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:29.956 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:30.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:30.005 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:30.074 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:30.075 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:30.075 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:30.075 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:30.076 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:30.076 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:30.076 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:30.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:30.077 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:30.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:30.077 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:30.078 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:30.078 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:30.079 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:30.079 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:30.080 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:30.080 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:30.081 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:30.081 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:30.082 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:30.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:30.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:30.083 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:30.084 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:30.084 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:30.084 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:30.085 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:30.085 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:30.086 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:30.086 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:30.086 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:30.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:30.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:30.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:30.088 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:30.088 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:30.088 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:30.089 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:30.089 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:30.089 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:30.089 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:30.090 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:30.090 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:30.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:30.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:30.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:30.091 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:30.091 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:30.092 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:30.092 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:30.092 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:30.092 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:30.093 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:30.093 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:30.093 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:30.093 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:30.094 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:30.094 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:30.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:30.095 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:30.095 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:30.095 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:30.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:30.096 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:30Z" 2022-07-18T04:08:30.096 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:30.097 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:30Z" } 2022-07-18T04:08:30.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:30.099 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:30.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:30.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:30.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2022-07-18T04:08:35.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:35.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:35.103 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:35.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:35.159 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:35.234 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:35.235 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:35.235 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:35.235 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:35.236 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:35.236 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:35.237 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:35.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:35.238 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:35.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:35.239 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:35.239 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:35.240 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:35.240 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:35.240 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:35.241 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:35.242 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:35.242 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:35.243 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:35.243 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:35.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:35.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:35.245 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:35.245 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:35.246 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:35.246 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:35.247 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:35.247 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:35.248 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:35.248 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:35.249 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:35.249 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:35.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:35.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:35.251 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:35.251 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:35.252 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:35.252 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:35.253 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:35.253 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:35.254 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:35.254 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:35.255 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:35.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:35.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:35.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:35.257 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:35.257 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:35.258 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:35.258 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:35.258 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:35.259 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:35.259 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:35.260 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:35.260 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:35.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:35.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:35.261 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:35.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:35.262 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:35.263 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:35.263 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:35.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:35.264 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:35Z" 2022-07-18T04:08:35.265 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:35.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:35Z" } 2022-07-18T04:08:35.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:35.268 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:35.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:35.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:35.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~286s 2022-07-18T04:08:40.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:40.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:40.274 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:40.330 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:40.330 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:40.397 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:40.397 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:40.398 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:40.398 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:40.398 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:40.398 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:40.399 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:40.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:40.399 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:40.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:40.400 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:40.400 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:40.400 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:40.401 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:40.401 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:40.401 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:40.401 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:40.402 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:40.402 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:40.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:40.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:40.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:40.403 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:40.403 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:40.403 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:40.404 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:40.404 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:40.404 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:40.404 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:40.405 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:40.405 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:40.405 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:40.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:40.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:40.406 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:40.406 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:40.407 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:40.407 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:40.408 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:40.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:40.408 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:40.408 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:40.409 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:40.409 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:40.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:40.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:40.410 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:40.410 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:40.411 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:40.411 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:40.411 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:40.411 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:40.412 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:40.412 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:40.412 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:40.412 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:40.413 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:40.413 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:40.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:40.413 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:40.414 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:40.414 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:40.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:40.414 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:40Z" 2022-07-18T04:08:40.415 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:40.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:40Z" } 2022-07-18T04:08:40.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:40.418 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:40.418 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:40.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:41.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~292s 2022-07-18T04:08:45.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:45.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:45.423 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:45.480 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:45.480 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:45.585 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:45.585 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:45.585 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:45.586 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:45.586 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:45.586 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:45.587 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:45.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:45.587 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:45.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:45.588 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:45.588 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:45.588 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:45.589 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:45.589 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:45.589 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:45.590 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:45.590 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:45.590 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:45.591 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:45.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:45.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:45.591 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:45.592 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:45.592 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:45.592 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:45.592 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:45.593 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:45.593 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:45.593 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:45.594 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:45.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:45.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:45.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:45.597 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:45.597 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:45.598 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:45.598 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:45.598 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:45.599 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:45.599 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:45.600 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:45.600 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:45.600 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:45.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:45.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:45.602 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:45.602 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:45.602 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:45.602 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:45.603 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:45.603 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:45.603 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:45.604 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:45.604 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:45.604 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:45.604 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:45.605 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:45.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:45.605 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:45.605 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:45.606 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:45.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:45.606 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:45Z" 2022-07-18T04:08:45.606 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:45.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:45Z" } 2022-07-18T04:08:45.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:45.610 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:45.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:45.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:46.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~297s 2022-07-18T04:08:50.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:50.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:50.616 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:50.668 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:50.669 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:50.758 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:50Z" 2022-07-18T04:08:50.780 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:50.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:50Z" } 2022-07-18T04:08:50.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:50.783 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:50.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:50.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:08:52.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~303s 2022-07-18T04:08:52.537 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-07-18T04:08:53.142 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2022-07-18T04:08:53.145 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2022-07-18T04:08:53.146 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2022-07-18T04:08:53.146 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2022-07-18T04:08:53.147 INFO:tasks.rgw.c1.client.0:Sent signal 15 2022-07-18T04:08:53.148 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2022-07-18T04:08:53.150 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T04:08:53.141+0000 7f7692cce700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 40715) UID: 0 2022-07-18T04:08:53.150 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T04:08:53.141+0000 7f7692cce700 -1 osd.0 64 *** Got signal Terminated *** 2022-07-18T04:08:53.150 INFO:tasks.ceph.c1.osd.0.smithi133.stderr:2022-07-18T04:08:53.141+0000 7f7692cce700 -1 osd.0 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-07-18T04:08:53.151 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T04:08:53.144+0000 7fbcdd1a9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 40721) UID: 0 2022-07-18T04:08:53.151 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T04:08:53.144+0000 7fbcdd1a9700 -1 osd.1 64 *** Got signal Terminated *** 2022-07-18T04:08:53.152 INFO:tasks.ceph.c1.osd.1.smithi133.stderr:2022-07-18T04:08:53.144+0000 7fbcdd1a9700 -1 osd.1 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-07-18T04:08:53.152 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T04:08:53.145+0000 7fec0c7cd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 40724) UID: 0 2022-07-18T04:08:53.152 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T04:08:53.145+0000 7fec0c7cd700 -1 osd.2 64 *** Got signal Terminated *** 2022-07-18T04:08:53.153 INFO:tasks.ceph.c1.osd.2.smithi133.stderr:2022-07-18T04:08:53.145+0000 7fec0c7cd700 -1 osd.2 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-07-18T04:08:53.154 INFO:tasks.ceph.c1.mon.a.smithi133.stderr:2022-07-18T04:08:53.145+0000 7f03a6952700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 40287) UID: 0 2022-07-18T04:08:53.154 INFO:tasks.ceph.c1.mon.a.smithi133.stderr:2022-07-18T04:08:53.145+0000 7f03a6952700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-07-18T04:08:53.155 INFO:tasks.rgw.c1.client.0.smithi133.stdout:2022-07-18T04:08:53.146+0000 7f854702d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 48172) UID: 0 2022-07-18T04:08:53.156 INFO:tasks.rgw.c1.client.0.smithi133.stdout:2022-07-18T04:08:53.146+0000 7f8557bab600 -1 shutting down 2022-07-18T04:08:53.157 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T04:08:53.147+0000 7f111176f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 40313) UID: 0 2022-07-18T04:08:53.157 INFO:tasks.ceph.c1.mgr.x.smithi133.stderr:2022-07-18T04:08:53.147+0000 7f111176f700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-07-18T04:08:55.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:55.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:08:55.787 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:08:55.841 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:08:55.841 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:08:55.912 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:08:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:08:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:08:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:08:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:08:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:08:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:08:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:08:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:08:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:08:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:08:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:08:55.920 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:55.920 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:55.921 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:08:55.921 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:55.921 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:55.922 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:55.922 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:08:55.923 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:55.923 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:55.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:55.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:55.925 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:08:55.926 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:55.926 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:55.927 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:08:55.927 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:55.928 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:55.928 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:55.928 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:08:55.929 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:55.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:55.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:55.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:08:55.931 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:08:55.931 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:08:55.932 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:08:55.932 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:08:55.933 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:08:55.933 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:08:55.934 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:08:55.935 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:08:55.935 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:08:55.936 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:55.936 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:08:55.937 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:08:55.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:55.938 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:08:55.938 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:08:55.939 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:08:55.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:08:55.940 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:08:55Z" 2022-07-18T04:08:55.940 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:08:55.942 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:08:55Z" } 2022-07-18T04:08:55.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:08:55.945 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:08:55.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:08:55.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:00.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:00.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:00.950 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:01.007 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:01.008 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:01.079 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:01.080 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:01.080 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:01.081 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:01.081 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:01.081 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:01.082 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:01.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:01.082 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:01.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:01.083 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:01.083 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:01.083 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:01.083 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:01.084 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:01.084 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:01.084 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:01.085 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:01.085 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:01.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:01.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:01.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:01.086 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:01.086 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:01.086 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:01.087 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:01.087 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:01.087 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:01.087 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:01.088 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:01.088 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:01.088 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:01.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:01.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:01.089 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:01.089 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:01.090 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:01.090 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:01.091 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:01.091 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:01.091 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:01.091 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:01.092 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:01.092 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:01.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:01.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:01.093 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:01.093 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:01.093 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:01.094 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:01.094 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:01.094 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:01.094 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:01.095 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:01.095 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:01.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:01.095 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:01.096 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:01.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:01.096 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:01.096 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:01.097 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:01.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:01.097 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:01Z" 2022-07-18T04:09:01.097 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:01.099 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:01Z" } 2022-07-18T04:09:01.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:01.100 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:01.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:01.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:06.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:06.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:06.105 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:06.161 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:06.162 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:06.235 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:06.235 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:06.235 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:06.236 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:06.236 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:06.236 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:06.236 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:06.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:06.237 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:06.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:06.238 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:06.238 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:06.238 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:06.239 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:06.239 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:06.239 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:06.240 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:06.240 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:06.240 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:06.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:06.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:06.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:06.241 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:06.241 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:06.242 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:06.242 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:06.242 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:06.242 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:06.243 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:06.243 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:06.243 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:06.243 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:06.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:06.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:06.244 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:06.244 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:06.245 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:06.245 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:06.245 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:06.245 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:06.246 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:06.246 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:06.247 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:06.247 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:06.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:06.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:06.248 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:06.248 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:06.248 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:06.248 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:06.249 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:06.249 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:06.249 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:06.249 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:06.250 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:06.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:06.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:06.251 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:06.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:06.251 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:06.251 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:06.252 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:06.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:06.252 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:06Z" 2022-07-18T04:09:06.252 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:06.254 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:06Z" } 2022-07-18T04:09:06.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:06.255 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:06.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:06.257 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:11.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:11.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:11.261 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:11.316 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:11.317 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:11.383 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:11.384 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:11.384 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:11.385 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:11.385 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:11.385 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:11.385 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:11.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:11.386 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:11.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:11.386 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:11.387 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:11.387 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:11.388 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:11.388 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:11.388 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:11.389 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:11.390 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:11.390 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:11.390 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:11.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:11.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:11.391 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:11.391 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:11.392 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:11.392 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:11.392 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:11.392 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:11.393 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:11.393 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:11.393 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:11.400 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:11.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:11.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:11.401 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:11.401 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:11.402 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:11.402 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:11.402 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:11.402 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:11.403 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:11.403 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:11.403 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:11.403 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:11.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:11.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:11.404 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:11.404 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:11.405 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:11.405 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:11.405 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:11.405 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:11.406 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:11.406 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:11.406 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:11.406 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:11.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:11.407 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:11.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:11.407 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:11.408 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:11.408 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:11.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:11.408 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:11Z" 2022-07-18T04:09:11.409 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:11.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:11Z" } 2022-07-18T04:09:11.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:11.412 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:11.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:11.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:16.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:16.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:16.416 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:16.471 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:16.472 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:16.535 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:16Z" 2022-07-18T04:09:16.552 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:16.553 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:16Z" } 2022-07-18T04:09:16.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:16.555 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:16.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:16.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:21.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:21.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:21.559 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:21.614 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:21.615 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:21.681 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:21.692 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:21.693 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:21.693 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:21.693 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:21.694 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:21.694 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:21.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:21.694 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:21.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:21.695 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:21.695 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:21.696 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:21.696 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:21.696 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:21.697 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:21.697 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:21.697 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:21.698 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:21.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:21.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:21.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:21.701 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:21.701 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:21.701 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:21.701 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:21.702 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:21.702 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:21.702 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:21.703 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:21.703 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:21.703 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:21.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:21.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:21.704 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:21.704 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:21.704 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:21.705 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:21.705 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:21.705 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:21.705 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:21.705 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:21.706 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:21.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:21.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:21.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:21.707 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:21.707 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:21.707 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:21.708 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:21.708 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:21.708 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:21.708 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:21.709 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:21.709 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:21.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:21.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:21.709 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:21.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:21.710 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:21.711 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:21.711 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:21.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:21.711 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:21Z" 2022-07-18T04:09:21.712 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:21.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:21Z" } 2022-07-18T04:09:21.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:21.714 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:21.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:21.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:26.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:26.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:26.718 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:26.772 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:26.850 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:26.851 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:26.851 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:26.851 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:26.851 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:26.852 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:26.852 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:26.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:26.853 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:26.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:26.853 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:26.853 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:26.854 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:26.854 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:26.854 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:26.855 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:26.855 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:26.855 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:26.855 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:26.856 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:26.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:26.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:26.856 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:26.857 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:26.857 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:26.857 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:26.857 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:26.858 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:26.858 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:26.858 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:26.859 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:26.859 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:26.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:26.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:26.860 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:26.861 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:26.861 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:26.861 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:26.861 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:26.862 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:26.862 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:26.862 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:26.862 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:26.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:26.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:26.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:26.863 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:26.864 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:26.864 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:26.864 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:26.865 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:26.865 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:26.865 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:26.865 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:26.866 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:26.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:26.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:26.867 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:26.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:26.868 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:26.868 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:26.868 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:26.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:26.869 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:26Z" 2022-07-18T04:09:26.869 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:26.870 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:26Z" } 2022-07-18T04:09:26.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:26.872 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:26.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:26.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:31.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:31.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:31.876 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:31.933 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:31.933 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:32.007 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:31Z" 2022-07-18T04:09:32.029 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:32.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:31Z" } 2022-07-18T04:09:32.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:32.033 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:32.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:32.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:37.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:37.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:37.039 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:37.096 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:37.096 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:37.168 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:37.169 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:37.169 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:37.169 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:37.170 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:37.170 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:37.170 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:37.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:37.171 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:37.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:37.171 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:37.172 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:37.172 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:37.172 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:37.172 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:37.173 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:37.173 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:37.173 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:37.174 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:37.174 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:37.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:37.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:37.175 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:37.176 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:37.176 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:37.176 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:37.176 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:37.177 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:37.177 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:37.177 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:37.177 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:37.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:37.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:37.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:37.179 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:37.179 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:37.180 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:37.180 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:37.181 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:37.181 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:37.182 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:37.182 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:37.183 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:37.184 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:37.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:37.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:37.185 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:37.186 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:37.186 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:37.187 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:37.187 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:37.188 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:37.188 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:37.190 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:37.190 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:37.190 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:37.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:37.191 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:37.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:37.191 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:37.192 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:37.192 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:37.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:37.192 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:37Z" 2022-07-18T04:09:37.193 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:37.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:37Z" } 2022-07-18T04:09:37.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:37.195 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:37.196 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:37.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:42.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:42.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:42.199 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:42.256 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:42.257 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:42.340 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:42.341 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:42.341 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:42.342 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:42.342 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:42.342 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:42.342 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:42.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:42.343 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:42.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:42.343 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:42.344 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:42.344 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:42.344 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:42.344 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:42.345 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:42.345 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:42.346 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:42.346 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:42.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:42.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:42.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:42.348 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:42.348 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:42.349 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:42.349 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:42.350 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:42.350 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:42.351 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:42.351 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:42.352 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:42.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:42.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:42.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:42.354 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:42.354 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:42.355 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:42.355 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:42.356 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:42.356 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:42.357 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:42.357 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:42.358 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:42.358 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:42.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:42.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:42.359 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:42.360 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:42.360 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:42.361 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:42.361 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:42.362 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:42.363 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:42.363 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:42.363 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:42.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:42.364 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:42.364 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:42.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:42.365 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:42.365 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:42.365 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:42.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:42.366 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:42Z" 2022-07-18T04:09:42.366 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:42.368 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:42Z" } 2022-07-18T04:09:42.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:42.369 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:42.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:42.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:47.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:47.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:47.373 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:47.429 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:47.430 INFO:teuthology.orchestra.run.smithi159.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:09:47.494 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-18T04:09:47.495 INFO:teuthology.orchestra.run.smithi159.stdout: "sync_status": { 2022-07-18T04:09:47.495 INFO:teuthology.orchestra.run.smithi159.stdout: "info": { 2022-07-18T04:09:47.496 INFO:teuthology.orchestra.run.smithi159.stdout: "status": "sync", 2022-07-18T04:09:47.496 INFO:teuthology.orchestra.run.smithi159.stdout: "num_shards": 4, 2022-07-18T04:09:47.497 INFO:teuthology.orchestra.run.smithi159.stdout: "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", 2022-07-18T04:09:47.497 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:47.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:47.499 INFO:teuthology.orchestra.run.smithi159.stdout: "markers": [ 2022-07-18T04:09:47.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:47.505 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 0, 2022-07-18T04:09:47.506 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:47.506 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:47.506 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113483.073958_74.1", 2022-07-18T04:09:47.507 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:47.507 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 1, 2022-07-18T04:09:47.507 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:47.507 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:04:43.073958Z", 2022-07-18T04:09:47.508 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:47.508 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:47.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:47.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:47.509 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 1, 2022-07-18T04:09:47.509 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:47.509 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:47.510 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "", 2022-07-18T04:09:47.510 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:47.510 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:47.511 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:47.511 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "0.000000", 2022-07-18T04:09:47.511 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:47.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:47.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:47.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:47.512 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 2, 2022-07-18T04:09:47.512 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:47.513 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:47.513 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658117082.891928_311.1", 2022-07-18T04:09:47.513 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:47.513 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:47.514 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:47.514 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T04:04:42.891928Z", 2022-07-18T04:09:47.514 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:47.514 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:47.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:47.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2022-07-18T04:09:47.515 INFO:teuthology.orchestra.run.smithi159.stdout: "key": 3, 2022-07-18T04:09:47.515 INFO:teuthology.orchestra.run.smithi159.stdout: "val": { 2022-07-18T04:09:47.516 INFO:teuthology.orchestra.run.smithi159.stdout: "state": 1, 2022-07-18T04:09:47.516 INFO:teuthology.orchestra.run.smithi159.stdout: "marker": "1_1658113606.573510_120.1", 2022-07-18T04:09:47.516 INFO:teuthology.orchestra.run.smithi159.stdout: "next_step_marker": "", 2022-07-18T04:09:47.516 INFO:teuthology.orchestra.run.smithi159.stdout: "total_entries": 0, 2022-07-18T04:09:47.517 INFO:teuthology.orchestra.run.smithi159.stdout: "pos": 0, 2022-07-18T04:09:47.517 INFO:teuthology.orchestra.run.smithi159.stdout: "timestamp": "2022-07-18T03:06:46.573510Z", 2022-07-18T04:09:47.517 INFO:teuthology.orchestra.run.smithi159.stdout: "realm_epoch": 2 2022-07-18T04:09:47.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:47.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-18T04:09:47.518 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2022-07-18T04:09:47.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:47.519 INFO:teuthology.orchestra.run.smithi159.stdout: "full_sync": { 2022-07-18T04:09:47.519 INFO:teuthology.orchestra.run.smithi159.stdout: "total": 1, 2022-07-18T04:09:47.519 INFO:teuthology.orchestra.run.smithi159.stdout: "complete": 1 2022-07-18T04:09:47.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-18T04:09:47.520 INFO:teuthology.orchestra.run.smithi159.stdout: "current_time": "2022-07-18T04:09:47Z" 2022-07-18T04:09:47.520 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-18T04:09:47.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c2173771-3b17-40d3-8c8b-0df64165bc30", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1658113483.073958_74.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-07-18T03:04:43.073958Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1658117082.891928_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T04:04:42.891928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1658113606.573510_120.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-07-18T03:06:46.573510Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-07-18T04:09:47Z" } 2022-07-18T04:09:47.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1658113483.073958_74.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-07-18T03:04:43.073958Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1658117082.891928_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T04:04:42.891928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1658113606.573510_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-07-18T03:06:46.573510Z', 'realm_epoch': 2}}] 2022-07-18T04:09:47.523 DEBUG:rgw_multi.tests:log_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.719799_131.1'} 2022-07-18T04:09:47.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1658113483.073958_74.1', 1: '', 2: '1_1658117082.891928_311.1', 3: '1_1658113606.573510_120.1'} 2022-07-18T04:09:47.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1658113606.719799_131.1 target=1_1658113606.573510_120.1 2022-07-18T04:09:52.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2022-07-18T04:09:52.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:52.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T04:09:52.534 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T04:09:52.582 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:09:52.583 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T04:14:52.587 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:14:52.582+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:19:52.587 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:19:52.582+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:24:52.587 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:24:52.583+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:29:52.589 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:29:52.584+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:34:52.589 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:34:52.584+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:39:52.590 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:39:52.585+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:44:52.591 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:44:52.585+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:49:52.591 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:49:52.587+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:54:52.591 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:54:52.586+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:59:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T04:59:52.597 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T04:59:52.588+0000 7f6639289540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T04:59:52.597 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T04:59:52.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2022-07-18T04:59:52.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T04:59:52.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-07-18T04:59:52.603 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-07-18T04:59:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T04:59:52.657 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T05:04:52.661 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:04:52.655+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:09:52.662 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:09:52.657+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:14:52.662 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:14:52.657+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:19:52.663 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:19:52.658+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:24:52.663 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:24:52.658+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:29:52.663 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:29:52.658+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:34:52.664 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:34:52.658+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:39:52.665 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:39:52.659+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:44:52.665 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:44:52.660+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:49:52.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T05:49:52.670 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T05:49:52.660+0000 7fef66fdf540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T05:49:52.670 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T05:49:52.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... FAIL 2022-07-18T05:49:52.675 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-46 2022-07-18T05:49:52.676 DEBUG:boto:path=/mlvank-46/ 2022-07-18T05:49:52.679 DEBUG:boto:auth_path=/mlvank-46/ 2022-07-18T05:49:52.680 DEBUG:boto:Method: PUT 2022-07-18T05:49:52.680 DEBUG:boto:Path: /mlvank-46/ 2022-07-18T05:49:52.681 DEBUG:boto:Data: 2022-07-18T05:49:52.681 DEBUG:boto:Headers: {} 2022-07-18T05:49:52.682 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T05:49:52.682 DEBUG:boto:Port: 8000 2022-07-18T05:49:52.683 DEBUG:boto:Params: {} 2022-07-18T05:49:52.684 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:49:52.685 DEBUG:boto:Token: None 2022-07-18T05:49:52.686 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:49:52.687 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:49:52.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:49:52.691 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:49:52.696 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:49:52.923 DEBUG:boto:Token: None 2022-07-18T05:49:52.924 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:49:52.925 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:49:52.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:49:52.928 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:49:52.930 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:49:54.027 DEBUG:boto:Token: None 2022-07-18T05:49:54.028 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:49:54.028 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:49:54.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:49:54.031 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:49:54.031 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:49:57.501 DEBUG:boto:Token: None 2022-07-18T05:49:57.502 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:49:57.503 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:49:57.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:49:57.505 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:49:57.506 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:50:05.250 DEBUG:boto:Token: None 2022-07-18T05:50:05.251 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:50:05.251 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:50:05.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:50:05.254 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:50:05.254 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:50:10.878 DEBUG:boto:Token: None 2022-07-18T05:50:10.879 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:50:10.880 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:50:10.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:50:10.882 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:50:10.883 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:50:11.650 DEBUG:boto:Token: None 2022-07-18T05:50:11.651 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:49:52 GMT /mlvank-46/ 2022-07-18T05:50:11.652 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A= 2022-07-18T05:50:11.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:49:52 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:0ZMjQZ2h341bdzT/z1YJnkYJX6A=', 'Content-Length': '0'} 2022-07-18T05:50:11.655 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:50:11.655 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:09.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... ERROR 2022-07-18T05:51:09.665 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-47 2022-07-18T05:51:09.666 DEBUG:boto:path=/mlvank-47/ 2022-07-18T05:51:09.666 DEBUG:boto:auth_path=/mlvank-47/ 2022-07-18T05:51:09.667 DEBUG:boto:Method: PUT 2022-07-18T05:51:09.667 DEBUG:boto:Path: /mlvank-47/ 2022-07-18T05:51:09.668 DEBUG:boto:Data: 2022-07-18T05:51:09.668 DEBUG:boto:Headers: {} 2022-07-18T05:51:09.669 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T05:51:09.670 DEBUG:boto:Port: 8000 2022-07-18T05:51:09.670 DEBUG:boto:Params: {} 2022-07-18T05:51:09.671 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:09.671 DEBUG:boto:Token: None 2022-07-18T05:51:09.672 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:09.673 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:09.675 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:09.676 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:10.027 DEBUG:boto:Token: None 2022-07-18T05:51:10.029 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:10.030 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:10.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:10.034 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:10.035 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:10.417 DEBUG:boto:Token: None 2022-07-18T05:51:10.418 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:10.418 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:10.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:10.420 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:10.421 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:12.299 DEBUG:boto:Token: None 2022-07-18T05:51:12.300 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:12.300 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:12.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:12.304 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:12.305 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:19.533 DEBUG:boto:Token: None 2022-07-18T05:51:19.534 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:19.535 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:19.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:19.537 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:19.538 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:29.396 DEBUG:boto:Token: None 2022-07-18T05:51:29.397 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:29.397 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:29.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:29.400 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:29.401 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:51:40.186 DEBUG:boto:Token: None 2022-07-18T05:51:40.187 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:51:09 GMT /mlvank-47/ 2022-07-18T05:51:40.187 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg= 2022-07-18T05:51:40.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:51:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2yT+7bpb8SkDyLT7X7dM5VM9Esg=', 'Content-Length': '0'} 2022-07-18T05:51:40.190 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:51:40.191 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:33.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ERROR 2022-07-18T05:52:33.510 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-48 2022-07-18T05:52:33.511 DEBUG:boto:path=/mlvank-48/ 2022-07-18T05:52:33.511 DEBUG:boto:auth_path=/mlvank-48/ 2022-07-18T05:52:33.512 DEBUG:boto:Method: PUT 2022-07-18T05:52:33.513 DEBUG:boto:Path: /mlvank-48/ 2022-07-18T05:52:33.513 DEBUG:boto:Data: 2022-07-18T05:52:33.514 DEBUG:boto:Headers: {} 2022-07-18T05:52:33.514 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T05:52:33.515 DEBUG:boto:Port: 8000 2022-07-18T05:52:33.515 DEBUG:boto:Params: {} 2022-07-18T05:52:33.516 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:33.517 DEBUG:boto:Token: None 2022-07-18T05:52:33.518 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:52:33.519 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:52:33.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:52:33.521 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:52:33.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:33.864 DEBUG:boto:Token: None 2022-07-18T05:52:33.865 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:52:33.866 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:52:33.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:52:33.868 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:52:33.870 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:35.292 DEBUG:boto:Token: None 2022-07-18T05:52:35.293 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:52:35.294 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:52:35.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:52:35.296 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:52:35.297 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:36.925 DEBUG:boto:Token: None 2022-07-18T05:52:36.926 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:52:36.927 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:52:36.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:52:36.929 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:52:36.930 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:42.407 DEBUG:boto:Token: None 2022-07-18T05:52:42.408 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:52:42.409 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:52:42.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:52:42.411 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:52:42.413 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:52:44.697 DEBUG:boto:Token: None 2022-07-18T05:52:44.698 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:52:44.698 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:52:44.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:52:44.701 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:52:44.701 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:53:09.973 DEBUG:boto:Token: None 2022-07-18T05:53:09.975 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:52:33 GMT /mlvank-48/ 2022-07-18T05:53:09.976 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI= 2022-07-18T05:53:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:52:33 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:2EEhpDuqOMmzMHsTCRymiRSiSOI=', 'Content-Length': '0'} 2022-07-18T05:53:09.980 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:53:09.981 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:09.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2022-07-18T05:54:09.986 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-49 2022-07-18T05:54:09.987 DEBUG:boto:path=/mlvank-49/ 2022-07-18T05:54:09.988 DEBUG:boto:auth_path=/mlvank-49/ 2022-07-18T05:54:09.988 DEBUG:boto:Method: PUT 2022-07-18T05:54:09.989 DEBUG:boto:Path: /mlvank-49/ 2022-07-18T05:54:09.989 DEBUG:boto:Data: 2022-07-18T05:54:09.990 DEBUG:boto:Headers: {} 2022-07-18T05:54:09.990 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T05:54:09.991 DEBUG:boto:Port: 8000 2022-07-18T05:54:09.991 DEBUG:boto:Params: {} 2022-07-18T05:54:09.992 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:09.992 DEBUG:boto:Token: None 2022-07-18T05:54:09.993 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:09.994 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:09.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:09.997 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:09.997 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:10.251 DEBUG:boto:Token: None 2022-07-18T05:54:10.252 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:10.253 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:10.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:10.256 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:10.258 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:10.317 DEBUG:boto:Token: None 2022-07-18T05:54:10.318 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:10.319 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:10.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:10.321 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:10.321 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:11.329 DEBUG:boto:Token: None 2022-07-18T05:54:11.330 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:11.331 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:11.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:11.333 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:11.334 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:17.836 DEBUG:boto:Token: None 2022-07-18T05:54:17.837 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:17.838 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:17.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:17.840 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:17.841 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:20.752 DEBUG:boto:Token: None 2022-07-18T05:54:20.753 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:20.754 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:20.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:20.756 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:20.757 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:43.885 DEBUG:boto:Token: None 2022-07-18T05:54:43.886 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:09 GMT /mlvank-49/ 2022-07-18T05:54:43.887 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA= 2022-07-18T05:54:43.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:09 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:LLRbSB/yENRmVYNpSQYkVmabWeA=', 'Content-Length': '0'} 2022-07-18T05:54:43.890 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:43.891 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:48.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2022-07-18T05:54:48.136 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-50 2022-07-18T05:54:48.136 DEBUG:boto:path=/mlvank-50/ 2022-07-18T05:54:48.137 DEBUG:boto:auth_path=/mlvank-50/ 2022-07-18T05:54:48.138 DEBUG:boto:Method: PUT 2022-07-18T05:54:48.139 DEBUG:boto:Path: /mlvank-50/ 2022-07-18T05:54:48.140 DEBUG:boto:Data: 2022-07-18T05:54:48.141 DEBUG:boto:Headers: {} 2022-07-18T05:54:48.142 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T05:54:48.142 DEBUG:boto:Port: 8000 2022-07-18T05:54:48.143 DEBUG:boto:Params: {} 2022-07-18T05:54:48.143 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:48.144 DEBUG:boto:Token: None 2022-07-18T05:54:48.144 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:54:48.145 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:54:48.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:54:48.149 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:48.149 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:48.374 DEBUG:boto:Token: None 2022-07-18T05:54:48.375 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:54:48.376 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:54:48.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:54:48.379 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:48.379 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:48.650 DEBUG:boto:Token: None 2022-07-18T05:54:48.651 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:54:48.653 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:54:48.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:54:48.656 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:48.657 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:49.308 DEBUG:boto:Token: None 2022-07-18T05:54:49.309 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:54:49.310 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:54:49.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:54:49.312 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:49.313 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:54:56.430 DEBUG:boto:Token: None 2022-07-18T05:54:56.431 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:54:56.431 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:54:56.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:54:56.434 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:54:56.434 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:55:10.540 DEBUG:boto:Token: None 2022-07-18T05:55:10.542 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:55:10.543 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:55:10.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:55:10.547 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:55:10.548 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:55:37.713 DEBUG:boto:Token: None 2022-07-18T05:55:37.714 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:54:48 GMT /mlvank-50/ 2022-07-18T05:55:37.715 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw= 2022-07-18T05:55:37.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:54:48 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:QJaDoAY/icASrEgr2ZVyC/BLQAw=', 'Content-Length': '0'} 2022-07-18T05:55:37.717 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:55:37.718 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:00.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2022-07-18T05:56:00.681 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=mlvank-51 2022-07-18T05:56:00.682 DEBUG:boto:path=/mlvank-51/ 2022-07-18T05:56:00.683 DEBUG:boto:auth_path=/mlvank-51/ 2022-07-18T05:56:00.684 DEBUG:boto:Method: PUT 2022-07-18T05:56:00.685 DEBUG:boto:Path: /mlvank-51/ 2022-07-18T05:56:00.686 DEBUG:boto:Data: 2022-07-18T05:56:00.687 DEBUG:boto:Headers: {} 2022-07-18T05:56:00.688 DEBUG:boto:Host: smithi133.front.sepia.ceph.com:8000 2022-07-18T05:56:00.688 DEBUG:boto:Port: 8000 2022-07-18T05:56:00.689 DEBUG:boto:Params: {} 2022-07-18T05:56:00.690 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:00.691 DEBUG:boto:Token: None 2022-07-18T05:56:00.692 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:00.693 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:00.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:00.696 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:00.698 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:01.382 DEBUG:boto:Token: None 2022-07-18T05:56:01.383 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:01.384 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:01.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:01.387 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:01.387 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:02.763 DEBUG:boto:Token: None 2022-07-18T05:56:02.764 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:02.765 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:02.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:02.768 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:02.768 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:03.251 DEBUG:boto:Token: None 2022-07-18T05:56:03.252 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:03.252 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:03.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:03.255 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:03.256 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:05.959 DEBUG:boto:Token: None 2022-07-18T05:56:05.960 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:05.961 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:05.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:05.964 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:05.964 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:12.225 DEBUG:boto:Token: None 2022-07-18T05:56:12.227 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:12.228 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:12.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:12.233 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:12.233 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:20.386 DEBUG:boto:Token: None 2022-07-18T05:56:20.387 DEBUG:boto:StringToSign: PUT Mon, 18 Jul 2022 05:56:00 GMT /mlvank-51/ 2022-07-18T05:56:20.389 DEBUG:boto:Signature: AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs= 2022-07-18T05:56:20.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 18 Jul 2022 05:56:00 GMT', 'Authorization': 'AWS 1ZE2DVR9X6CB66IYMJG4:1Q87zejOHuy7ojnzUoVHrnvxiWs=', 'Content-Length': '0'} 2022-07-18T05:56:20.392 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T05:56:20.393 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T05:56:39.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... ERROR 2022-07-18T05:56:39.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T05:56:39.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T05:56:39.348 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T05:56:39.401 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T05:56:39.402 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T06:01:39.406 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:01:39.399+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:06:39.406 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:06:39.400+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:11:39.407 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:11:39.401+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:16:39.406 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:16:39.401+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:21:39.409 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:21:39.402+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:26:39.409 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:26:39.402+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:31:39.407 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:31:39.403+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:36:39.409 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:36:39.403+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:41:39.410 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:41:39.404+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:46:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T06:46:39.419 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:46:39.406+0000 7ff37714e540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:46:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T06:46:39.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2022-07-18T06:46:39.426 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-mlvank-1 2022-07-18T06:46:39.428 DEBUG:boto:Method: POST 2022-07-18T06:46:39.429 DEBUG:boto:Path: / 2022-07-18T06:46:39.430 DEBUG:boto:Data: 2022-07-18T06:46:39.431 DEBUG:boto:Headers: {} 2022-07-18T06:46:39.432 DEBUG:boto:Host: smithi133.front.sepia.ceph.com 2022-07-18T06:46:39.433 DEBUG:boto:Port: 8000 2022-07-18T06:46:39.434 DEBUG:boto:Params: {'RoleName': 'roles-mlvank-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-07-18T06:46:39.435 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:46:39.437 DEBUG:boto:Token: None 2022-07-18T06:46:39.439 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064639Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:46:39.440 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064639Z 20220718/front/smithi133/aws4_request dfab734d2e9bd340a3a6babe3b9a29ee12797ded31526bddabc9bffde4c1d505 2022-07-18T06:46:39.441 DEBUG:boto:Signature: 6cb9db01da23b0215249de90616874df8fe61a0d56a3fc77b9269f0b2d1ed642 2022-07-18T06:46:39.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064639Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=6cb9db01da23b0215249de90616874df8fe61a0d56a3fc77b9269f0b2d1ed642', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:46:39.446 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:46:39.448 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:46:39.688 DEBUG:boto:Token: None 2022-07-18T06:46:39.689 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064639Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:46:39.689 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064639Z 20220718/front/smithi133/aws4_request dfab734d2e9bd340a3a6babe3b9a29ee12797ded31526bddabc9bffde4c1d505 2022-07-18T06:46:39.690 DEBUG:boto:Signature: 6cb9db01da23b0215249de90616874df8fe61a0d56a3fc77b9269f0b2d1ed642 2022-07-18T06:46:39.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064639Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=6cb9db01da23b0215249de90616874df8fe61a0d56a3fc77b9269f0b2d1ed642', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:46:39.693 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:46:39.694 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:46:40.527 DEBUG:boto:Token: None 2022-07-18T06:46:40.528 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064640Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:46:40.529 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064640Z 20220718/front/smithi133/aws4_request a22aa2f8423cf1c5121b48c0422ba37ebdf22395dfac50ca81886cf3bc02d000 2022-07-18T06:46:40.529 DEBUG:boto:Signature: ee7fc28ebc31fc24a384051a18aa95c2ee9bbab87aec1b3c08bf39625bb1ab09 2022-07-18T06:46:40.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064640Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=ee7fc28ebc31fc24a384051a18aa95c2ee9bbab87aec1b3c08bf39625bb1ab09', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:46:40.532 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:46:40.532 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:46:41.993 DEBUG:boto:Token: None 2022-07-18T06:46:41.995 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064641Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:46:41.996 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064641Z 20220718/front/smithi133/aws4_request 9a92751b99efe0ad45086a2d9c48ec5b2cb4e0d6f57505fbcb8c1cdc30e05e5b 2022-07-18T06:46:41.997 DEBUG:boto:Signature: 8ff5322708eb28da121612d39f7a68c6d360f76a419eb840923ca075efaea7ad 2022-07-18T06:46:41.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064641Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=8ff5322708eb28da121612d39f7a68c6d360f76a419eb840923ca075efaea7ad', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:46:42.000 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:46:42.001 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:46:43.305 DEBUG:boto:Token: None 2022-07-18T06:46:43.307 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064643Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:46:43.308 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064643Z 20220718/front/smithi133/aws4_request f84a401f4ad7cb377abd896d35b45b0ee244bcd90fef3a755eb9a6567b587637 2022-07-18T06:46:43.309 DEBUG:boto:Signature: 082a4e32be631af57779d787eb7041389a5f4f15c453fcc7434874501acb466b 2022-07-18T06:46:43.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064643Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=082a4e32be631af57779d787eb7041389a5f4f15c453fcc7434874501acb466b', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:46:43.312 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:46:43.314 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:46:59.150 DEBUG:boto:Token: None 2022-07-18T06:46:59.152 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064659Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:46:59.154 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064659Z 20220718/front/smithi133/aws4_request cec954d0adae5f98bddb82975edf66867ee2d60166c5597f350507d049f1cd94 2022-07-18T06:46:59.155 DEBUG:boto:Signature: 008e0ca08b87bc3b23940ea258c02494e5c3b37a9f9228cd394328db56743974 2022-07-18T06:46:59.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064659Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=008e0ca08b87bc3b23940ea258c02494e5c3b37a9f9228cd394328db56743974', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:46:59.159 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:46:59.160 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:47:06.510 DEBUG:boto:Token: None 2022-07-18T06:47:06.512 DEBUG:boto:CanonicalRequest: POST / host:smithi133.front.sepia.ceph.com:8000 x-amz-date:20220718T064706Z host;x-amz-date 4218cd3e86379f786bc5d88455833bb0b72fde7e0ad3915aae549de733a6703d 2022-07-18T06:47:06.513 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220718T064706Z 20220718/front/smithi133/aws4_request 3ee0c28fcbbfafeea6278f325896e741d27306d82e1e0a4ef3f761efb7160a46 2022-07-18T06:47:06.514 DEBUG:boto:Signature: 66172d11398e64c7e029ad292af26d6823a61000b5be160e989744f7eed87634 2022-07-18T06:47:06.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220718T064706Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1ZE2DVR9X6CB66IYMJG4/20220718/front/smithi133/aws4_request,SignedHeaders=host;x-amz-date,Signature=66172d11398e64c7e029ad292af26d6823a61000b5be160e989744f7eed87634', 'Host': 'smithi133.front.sepia.ceph.com:8000'} 2022-07-18T06:47:06.516 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-07-18T06:47:06.517 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-07-18T06:47:18.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2022-07-18T06:47:18.385 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2022-07-18T06:47:18.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T06:47:18.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T06:47:18.391 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T06:47:18.444 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T06:47:18.445 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T06:52:18.448 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:52:18.442+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T06:57:18.447 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T06:57:18.443+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:02:18.449 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:02:18.443+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:07:18.448 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:07:18.444+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:12:18.450 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:12:18.444+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:17:18.449 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:17:18.445+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:22:18.450 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:22:18.444+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:27:18.450 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:27:18.445+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:32:18.451 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:32:18.446+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:37:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T07:37:18.461 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:37:18.446+0000 7f5460d80540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:37:18.461 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T07:37:18.463 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2022-07-18T07:37:18.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T07:37:18.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T07:37:18.467 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T07:37:18.516 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T07:37:18.517 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T07:42:18.520 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:42:18.515+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:47:18.519 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:47:18.516+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:52:18.520 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:52:18.516+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T07:57:18.521 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T07:57:18.517+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:02:18.521 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:02:18.518+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:07:18.521 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:07:18.517+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:12:18.523 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:12:18.518+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:17:18.523 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:17:18.518+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:22:18.523 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:22:18.519+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:27:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T08:27:18.528 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:27:18.519+0000 7f155cf7d540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:27:18.529 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T08:27:18.531 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2022-07-18T08:27:18.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T08:27:18.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T08:27:18.534 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T08:27:18.582 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T08:27:18.583 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T08:32:18.586 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:32:18.582+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:37:18.587 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:37:18.581+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:42:18.587 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:42:18.582+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:47:18.589 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:47:18.583+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:52:18.588 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:52:18.584+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T08:57:18.588 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T08:57:18.585+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:02:18.589 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:02:18.584+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:07:18.589 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:07:18.585+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:12:18.589 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:12:18.586+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:17:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T09:17:18.594 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:17:18.586+0000 7f1cbb6eb540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:17:18.595 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T09:17:18.597 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2022-07-18T09:17:18.598 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2022-07-18T09:17:18.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2022-07-18T09:17:18.600 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2022-07-18T09:17:18.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T09:17:18.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T09:17:18.603 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T09:17:18.653 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T09:17:18.653 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T09:22:18.657 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:22:18.652+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:27:18.657 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:27:18.654+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:32:18.657 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:32:18.654+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:37:18.658 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:37:18.654+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:42:18.659 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:42:18.655+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:47:18.659 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:47:18.655+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:52:18.660 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:52:18.656+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T09:57:18.660 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T09:57:18.657+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:02:18.661 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:02:18.657+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:07:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T10:07:18.668 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:07:18.657+0000 7f2cfdf0c540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:07:18.669 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T10:07:18.673 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2022-07-18T10:07:18.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T10:07:18.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T10:07:18.677 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T10:07:18.733 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T10:07:18.734 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T10:12:18.737 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:12:18.733+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:17:18.738 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:17:18.733+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:22:18.738 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:22:18.733+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:27:18.738 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:27:18.734+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:32:18.739 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:32:18.735+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:37:18.739 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:37:18.736+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:42:18.740 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:42:18.736+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:47:18.740 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:47:18.737+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:52:18.741 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:52:18.738+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:57:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T10:57:18.745 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T10:57:18.738+0000 7f27d95b7540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T10:57:18.746 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T10:57:18.748 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2022-07-18T10:57:18.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T10:57:18.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T10:57:18.752 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T10:57:18.802 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T10:57:18.802 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T11:02:18.806 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:02:18.804+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:07:18.806 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:07:18.804+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:12:18.807 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:12:18.805+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:17:18.807 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:17:18.805+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:22:18.808 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:22:18.805+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:27:18.812 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:27:18.806+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:32:18.809 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:32:18.806+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:37:18.811 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:37:18.806+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:42:18.811 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:42:18.807+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:47:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T11:47:18.816 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:47:18.807+0000 7f23b9fd5540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:47:18.817 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T11:47:18.819 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2022-07-18T11:47:18.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T11:47:18.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T11:47:18.823 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T11:47:18.874 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T11:47:18.875 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T11:52:18.878 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:52:18.873+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T11:57:18.879 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T11:57:18.874+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:02:18.878 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:02:18.874+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:07:18.879 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:07:18.875+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:12:18.879 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:12:18.875+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:17:18.879 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:17:18.876+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:22:18.880 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:22:18.876+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:27:18.880 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:27:18.877+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:32:18.880 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:32:18.876+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:37:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T12:37:18.886 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:37:18.877+0000 7fab03379540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:37:18.887 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T12:37:18.889 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2022-07-18T12:37:18.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T12:37:18.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T12:37:18.893 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T12:37:18.945 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T12:37:18.946 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T12:42:18.949 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:42:18.945+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:47:18.949 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:47:18.946+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:52:18.949 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:52:18.946+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T12:57:18.950 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T12:57:18.947+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:02:18.951 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:02:18.947+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:07:18.951 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:07:18.948+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:12:18.951 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:12:18.947+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:17:18.953 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:17:18.949+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:22:19.117 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:22:18.949+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:27:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T13:27:18.963 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:27:18.949+0000 7f888c0e3540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:27:18.963 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T13:27:18.966 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2022-07-18T13:27:18.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T13:27:18.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T13:27:18.970 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T13:27:19.026 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T13:27:19.026 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T13:32:19.030 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:32:19.026+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:37:19.030 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:37:19.026+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:42:19.031 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:42:19.027+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:47:19.031 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:47:19.026+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:52:19.031 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:52:19.028+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T13:57:19.032 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T13:57:19.028+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:02:19.033 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:02:19.029+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:07:19.033 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:07:19.029+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:12:19.033 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:12:19.029+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:17:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-18T14:17:19.039 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:17:19.029+0000 7f5f4361a540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:17:19.039 INFO:teuthology.orchestra.run.smithi133.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-07-18T14:17:19.042 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2022-07-18T14:17:19.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T14:17:19.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-07-18T14:17:19.046 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-07-18T14:17:19.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2022-07-18T14:17:19.095 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2022-07-18T14:22:19.097 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:22:19.094+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:27:19.098 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:27:19.094+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:32:19.099 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:32:19.095+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:37:19.100 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:37:19.095+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:42:19.100 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:42:19.096+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:47:19.101 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:47:19.096+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:52:19.113 INFO:teuthology.orchestra.run.smithi133.stderr:2022-07-18T14:52:19.096+0000 7fc51f6b8540 0 monclient(hunting): authenticate timed out after 300 2022-07-18T14:54:58.836 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-18T14:54:58.842 DEBUG:teuthology.task.console_log:Killing console logger for smithi133 2022-07-18T14:54:58.845 DEBUG:teuthology.task.console_log:Killing console logger for smithi159 2022-07-18T14:54:58.845 DEBUG:teuthology.exit:Finished running handlers